Ah ok, so it's that whole stuff with Venam after the second badge where I couldn't find her and then checked all kinds of stuff until I ended up in Goldenleaf.
I should have realized I messed up and forgot something but at least I still found a bug lol.
Welp, I guess I'll have to load one of my save files from before the empty town, catch up with that missing badge and replay that part with the castle since I can't reach Gearen from where I am right now.
E: Thank God, I found that out, I would have missed a pretty important and cool part of the game otherwise