Looking a bit further into it, I don't think that was the actual problem, but the tile that triggers the event with him can be avoided by walking down from where I am in the picture., letting you into the village without it triggering. Sorry for the confusion. I'll also attach my save if you want to look into it.
Game.rxdata