It's not that it's necessary to care about IVs and EVs later on, but once you get the chance to do those things it'll make the game significantly more relaxed later on.
To me, the early game was the hardest part, because you have so little things at your disposal.