The problem with leaving is you can never be sure the door will still be open when you return.
Sam knows about the doors.
Or at least, he thought he did. They always seem to surprise him though, slamming shut just when he begins to think things will be different. (But they never are)
Sometimes, what he thought was a window turns out to be a door, as well. He just wanted to look, he just wanted a taste… but that window had been an open door. When he looked back, it was already closed.
Sam is surrounded by doors. He's never quite sure which one is the right one, but he'll open them all if he has to.
When he opens one door, others close. Behind another door, he might find the key to those locked doors… or he just might close some more.
He worries one day he'll open the door that permanently locks the rest.
He thought he'd already been through that door. Turns out he had a Get-Out-Of-Jail-Free card.
Without the free part, of course. Reopening that door wasn't free. (Because nothing is)
The people he loves have the annoying habit of taking on door-like qualities. Sometimes he has the key. Other times, he picks the lock.
Then there are the times when he can't get in at all. He just knocks until they open. (Sometimes there's an ass-whooping on the other side. He doesn't mind. At least he's in.)
He knows he can be a bit of a door too. He fears that being too open will only close more doors. He tries though, for Dean. Dean doesn't like knocking. He just kicks doors open. Sam knows that it hurts them both. (He can't help but close himself every once in a while, though. Maybe he's hoping Dean still wants in.)
Sam understands the reasoning behind doors. He really does. That doesn't mean he likes them, not at all. If he had his way, there would be no doors.
He's too used to looking back and finding them closed. That hurts.
Especially the ones he didn't close himself.