Taka is a willing member of a terrorist organization that has killed at minimum hundreds, and probably well into the thousands of people. You know the people in Jasper and Beryl who died? Taka killed them. Oh he might have whined about it, or felt bad, but he did it. He activated the machine that killed them all. What's more, But its worse than that, since he knows what he's doing is wrong, and doesn't care enough to stop until he's forced to.
Cal meanwhile literally murders Kiki's Medicham in front of her. Oh, but he totes sabotaged the machine that was going to blow up the volcano. Despite there being no reason to think that he would know how to do that, or did it competently. He also couldn't be bothered to warn anyone in case something went wrong. He's definitely a better person than Taka is, or at least less guilty (and does have the decency to turn on Team Meteor of his own accord unlike a certain Chatot wielding person), but that doesn't absolve him of anything.
Fern though, like I said before. He's a loser. He hasn't accomplished anything in the entire game. Yeah he willingly joined Team Meteor, but lots of people have done that. The one "mission" he was put on after joining Meteor was a complete farce that ended with him and indeed the entire group humiliated by the player and the lowest ranked gym leaders. He hasn't managed to succeed at anything all game, and I don't see his track record changing whatsoever. He's a jerk, but so far he's also pretty much harmless due to his own incompetence and stupidity.
Fern's not even in the top five horrible people in the game, probably not even top ten. Like I said before, hating him is way, way more than he deserves. Ignore him, and it would both infuriate him far more, and be far closer to what he deserves.