Sanji doesn't like to notice his lack of grace. But as a man who has been entirely self-aware from the first sign of his voice changing and hair appearing in places it hadn't been before, he can't help but realize that his poise is somewhat lacking these days.
This has never been a problem before. Back on Baratie, Sanji was able to balance his love of ass-kicking with his love for the fairer sex, appearing as a constant gentleman even as he ground unruly assholes into the floorboards. His fellow cooks never could have achieved Sanji's level of charm (not to mention his looks) and so didn't even bother trying to crowd his territory. He had been the undisputed cock of the walk.
He could almost hold the same position here on the Going Merry, but for a significant lack of pride on the part of the other men around who could possibly contend with the cook. This seemingly innocuous observation is a severe problem for Sanji.
Luffy's goal is to be king of the pirates. Zoro's goal is to be the greatest swordsman in the world. Lofty aspirations, to be sure, but at least they're committed.
So how can they just live and let live when they lose a fight? Sanji refuses to lose. Even if he ends up as a puddle of flesh and organs, Sanji will win whatever fight he takes part in. If he's a pile of blood and guts, then all that will remain of his opponent is a small scorch mark on the ground where the dude had spontaneously combusted from the force of Sanji's feet. The sword-waving idiot and the captain (who was also a bit of an idiot, but much less so than that marimo-brain) often fight right up till they're on Death's doorstep, but otherwise accept their defeats, if not easily, than at least with more dignity than Sanji could muster in the same situation.
It is in Sanji's nature to be competitive. It just so happens that he has never before needed to push himself to win.
And the worst part is that even when he wins, he still feels like a loser in the face of Luffy's unchecked optimism and Zoro's tacit resolve. They apparently do not see a need to be the best all the time like Sanji does, which puts him in the position of being the pissy guy who argues all the time. And this is not an attractive quality. He directly blames the captain and his first mate for his lack of progress with any ladies who board their ship.
But he can't yell at them about it, because Luffy would just laugh at him (and probably not understand the question) and the shitty swordsman would probably fall asleep. Talk about grace under pressure… damn it.
Sanji comforts himself by imagining Luffy and Zoro in the future, triumphant, one a king, the other a champion. Then he imagines himself kicking their sorry butts off the ship and striding back into the kitchen to complete a masterpiece of a meal (thanks in part to All Blue's bountiful supply of goodies) that they don't get to eat.
Because everyone knows that it's only when you've beaten the best of the best that you can call yourself Number One.