Well you can get into the cave via the exit where you, Erin, Flora, and Florin got out but if you can't find that then head into Darchlight Woods and find the field of purple flowers, go to the area with a million cliff jumps, and then fall down through one of those holes.
Oof, someone beat me.