Each time I have to deal with the Elite 4 in each game, whatever the team I'm building, there is always a member who frightened me a lot, or at least one of his or her Pokemon. For me, it qualifies as being the "hardest", or at least the most annoying, Elite 4 that I absolutely want to crush.
First Generation : Lorelei. Although she is supposed to be the weakest, her Jynx can give me trouble with the sleep and double stab combo. And I'm not talking about the Lapras, which can be a big problem with his bulk and attacks.
But nonetheless, sometimes, it's like taking a candy from a baby.
Second Generation : I could say Koga, who frightened me as a kid with his Crobat, but it's nothing compared to Karen and the mighty Houndoom! Especially in 4th gen. Bonus points for the Gengar : besides the terrible movepool, Destiny Bond is something worth to pay attention, especially in Nuzlockes.
Third Generation : Surprisingly, I would say Phoebe. Despite lacking offense Power, since i'm only relying on brute force, her ghosts' good defense and status moves are always annoying.
But only annoying. Nothing special about this Elite 4 as a whole. But it's still worth noting.
Fourth Gen : I hate Lucian! And his flipping Alakazam! One of the most powerful Psychic Pokemon, all generations taken in account. Plus a good coverage. That's always something I have to eliminate quickly.
Fifth Gen : As the first Generation where all Elites have Pokemon equally leveled (and also with good quality Pokemon), my choice is basically hard to make. I will give bonus to Marshall, for his annoying Sawk, not only a good sweeper, but also able to survive a hit. Fortunately, he will spam Full restore, leaving none for at least conkeldurr.
Sixth Gen : This one might be easy, due to the power of grinding in these games. But I think Siebold is not to be underestimated. He has better quality mons compared to the rest of the Elite 4, with good movepools and coverage.
As a subsequent note, I think most of the Elite pre 5th Gen would be more dangerous if the actual system (AKA Elite 4 on an equal standing of levels) was applied. What do you think?