You can't retaliate to bullying if you've used your Q? You seem to have forgotten about the WE combo with the E having the same range if cast at the edge of the indicator, aka a decent response to any mage which tries to jump on you after you farm.
His base damage at level 4 isn't good enough to encourage the idea of abandoning consistent farming; the base numbers are decent if you land an E or 2, but without the AP bonuses and the sufficient mana (equilibrium doesn't consistently ready up Q if you've used up your mana) you're not going to be able to trade at all in the early game without backing on several occasions; it's a rocky road whether or not you're harassing them in the first place.
I'd disagree that trading is worse in mid lane; an Orb of Deception or 2 flying at you is easier to face than a Riven facerolling over you or a Jax jumping on top of your face. The enemy has available harass in both situations, but one is harder to deal with.
For clarification, I'd like to state my typical strategy as Veigar in the midlane; I focus my Q on farming the minions, and then immediately retaliate with a W+E combo when they try to jump on me because of it's usage; the trade is sufficient even if they poke a few times afterwards since his early game damage isn't terrible, as you've stated. Of course, this means you don't have any way to fight properly before level 3 but that applies regardless of whether or not you're playing aggressively, and Q at level 1 does remarkably low damage anyway, making it redundant. Your first few levels against assassins are survivable anyway, since they can't really poke (unless they're Zed) consistently unless they go ham on you. As a result of this situation for a little while, which will likely involve a secured kill if the jungler has made a solid gank since E is an excellent followup, I then have stockpiled a decent low double-digit number of bonus AP and can proceed to now spamming the EWQ combo as harass, which not only heavily deters them due to the fact that half their health has disappeared, but also allows you to farm for a small lapse of time since their face is wrecked if you bust out a RQ Ignite in the meantime, meaning they will stay away unless they're an assassin who can confidently secure a kill on you; meaning that as a result of my strategy before level 6, I have put myself in a position where I can freely harass them because of my small AP stockpile which can then accelerate due to secured kills and some continued farming, meaning I am vastly superior. Assassins are an exception, of course, but if you're facing one of them then it's likely you won't be doing so well anyway.