The first time he saw her, he thought the same as his brother, she was NORMAL. She wasn't really eye-catching, she had washboard-flat chest and she was POOR.
But because she was poor, and had apparently a lot of guts, he and his brother decided that she'll be their newest toy. They played a lot with her naivety and PLAINESS. Always, they were awe-struck by her apparent indifference. Then, they found out that she can tell which is which between them, without even batting an eyelash.
Kaoru then knew that she was the one; the very first person to have fully stepped into their world. Their world that only consists of "us" and "apart from us". She was what they will suppose as an enigma. The light of their once pitch black world since she began to distinguish them.
He then decided to quietly observe. He figured that he might as well know who his competitors are; to his utter surprise, the ALL like Haruhi; in the romantic sense of the word; but his heart broke knowing that Hikaru, his precious brother, also likes her; though that should've been obvious, seeing as they are twins.
He was faced with a dilemma. Who's the most important for him? Hikaru or Haruhi? After a lot of struggles ( i.e troubling Haruhi, Huni-sempai and Mori-senpai) he finally found an answer. Hikaru was ALWAYS more important for Kaoru.
Though he undoubtedly likes Haruhi, no one can replace Hikaru in his heart. Hikaru, after all, is a part of him. Scientifically, that is. They were one, separated by birth.
So he dropped out of the contest, to support Hikaru, even if it'll hurt. As it were, Hikaru might have won. But he didn't.
Haruhi had decided to fall for Tono. That stupid, idiotic, naïve yet kind and charismatic Tono. The same Tono which saved Hikaru and him. The twins were forced to reassess everything. Tono likes Haruhi. Haruhi likes Tono. There were no place for them in the equation.
They had to surrender. For the happiness of the girl they like and the man who saved them. No matter how much it hurts, they were more important.