Varies by playthrough, but I think Charlotte's been the hardest for me. You can't really use her field against her (aside from a Blizzard user or Flash Fire mon) so you really have to plan well.
Noel used to be tough for me too, but nowadays I just lead with a Sludge Wave user and that nerfs his team quite a bit.
Looking ahead, I think Amaria's going to be really hard to beat assuming we fight her on a water field - to the point where Sludge Wave may be the only thing that stops me from getting completely haxed by her.
Special mention to Julia, who's either super easy or super hard depending on the starter you picked