Gym battles are among the best things of this game in my opinion.
They are hard, yep, but this is a good thing we just can't run the game like we do for main games. You have to think about strategies and capture a lot of different pokemon to counter the leader and the field effects.
Indeed this could take some time, but we have time, this game is not going to be fully done for years so this is not a problem. Items for EV training are available quite soon and you have a lot of options to exp your pokemon quickly enough.
It's great to have challenge during the whole game, not just post E4, imo.