There were several things that aggravated Batman. Most of these things were villains such as Joker, Scarecrow, and Bane. There were reasons these villains aggravated them.
For instance, Scarecrow. The man was a threat, a very serious one and his fear gas was no laughing matter. There were several times he'd been hit with it and every time it had a different component, so there was absolutely no way for him to become immune. It also meant creating a new antidote every few months for the inevitable breakout and takeover scheme.
Bane, arguably was the worse threat, he had in fact broke Batman's back, and caused the man other severe injuries from times when they had fought. There were countless times when Batman had to limp into the Batcave, needing immediate attention from Alfred and Leslie.
While Bane may have been the biggest threat physically, it was no question who was the biggest threat mentally. Obviously Joker was not mentally sane, but that's what made him the most difficult foe. There was no way to predict what he wanted, no way to keep him in jail. Joker was a maniac, bent on murdering and hurting as many people as possible. He had no master plan except for watching the world burn.
Now, there were several things that annoyed Bruce Wayne. Most of those things were business related. One being his secretary, another paper work, and the worst of them all, board meetings.
His secretary was a bother because she always flirted, even when Bruce expressly and nicely asked her not to.
Paperwork gave him the worst headaches, and he sometimes preferred a punch to the head from Bane rather than paperwork.
Don't even get him started on board meetings. Sitting for hours on end with a group of men who did nothing but argue the whole time. It was taxing and annoying.
But the one thing that both of his persona's hated was someone hurting his little bird. His Robin. Anyone that did so would face the wrath of Daddy Bats.