He dreads the silence in the TARDIS. He fears it in fact, because it gives his mind too much time to wonder about them. The Doctor doesn't want to think about it, he just wants to detach himself from them and move on. He can't and that hurts the most. He loves Rose, he really does and he knows that even when he regenerates again she'll still own a tiny piece of him. He doesn't want to think about leaving her on that beach with his other self so that they could build a life together.
He didn't realize that he was avoiding mirrors at first until he was stuck chasing a Renbog in a carnival hall of mirrors. Every time he caught sight of his own face and his own hair, he saw his double. He saw his other self with his Rose. He saw them holding hands, laughing and talking. In his own face he saw the man he hated most and envied most. He didn't want to imagine that man with his memories and face making love to the one he loved. He didn't want to imagine what their morning routine was in their own little home. The Doctor didn't want to imagine the feel of her lips against his, something he had only felt once in this body if you counted Cassandra. He didn't want to guess how many times his other-self felt her kiss every day.
His own face served as a constant reminder of what he had lost, no what he had sacrificed. He didn't want to imagine what this face looked like as it grew old in time with hers. He didn't want to imagine what features from him, their children might have. He didn't want to see in his mind a little girl with Rose's nose and hair and his eyes. The Doctor didn't want to think about any of it. He didn't want to wonder, he didn't want to ask and he really didn't want to imagine what kind of life he is having with her.