First was Azazel.
And it always came down to a demon, in the end, whether a true demon or not, and Azazel must have been first even though Sam didn't know his name properly until he was long dead. Growing up, he was not only a Hunter's kid, he was John Winchester's kid, and often referred to as 'the poor sod dragged along headfirst into the supernatural mess', and to top it all off, he had demon blood in him. Azazel was first because Azazel mattered first—to him and everyone, even though technically Meg was there first, and Lucifer was definitely there before anything else.
Sam was the rebel, the smart one. Standford. Jess. Poor, amazing Jess. He never healed from her death. And then there were the others with powers, with the urge to get high on demon blood. There was Jake, the knife in his back, and the two-day jaunt in Hell which Sam, copying Dean's infamous example, stuffed away somewhere in his soul and denied any remembrance of it, while it plagued him for a year, until about the time when Dean got dragged to hell.
Second was Ruby.
She was…different. Promises and nights and days of demon blood bliss and possibly, possibly, love. He had even chosen her over Dean, his own brother, and that meant she mattered to him. Given, they were trying to defeat Lilith, but Lilith wasn't the main cause. Lilith didn't matter; Ruby did. He had even walked out the door on his brother, signifying something deep—yes, Sam had loved Ruby. But the love for her was far outweighed by the hate for the Devil, even before Sam had met him.
Third was Meg.
Though Meg could have been second, or third in a different list. She could have been first if Azazel had only mattered to John and Dean, but Jess happened, and certain things happened with Meg and the demon remained alive, snarky, and somewhere between helping the Winchesters and trying to drag them to the depths of Hell. Sam was repulsed and intrigued as Meg left scar after mental scar on him. She signified, to Sam, the beginning of the end, as she was there when he first left Dean—before Ruby, even, though did she matter as much as Ruby before she killed Jo, making her third on the list, after Ruby?
Fourth was Lucifer.
If Sam was messed up before, well…nothing could compare. Though Lucifer could have been first, could have been Sam's 'match made in heaven' (literally) if Sam acknowledged it, and if he acknowledged that Lucifer had been with him from before the beginning. But no, he only came after the Azazel mess, the Meg mess (or so it seemed), and as a direct cause of the Ruby mess, from which Sam was still recovering when Lucifer entered his line of vision. Walls were broken and Cages were entered and mind merged, melded, and broke with imprints left that would never be forgotten.
Fifth was Sam.
Soulless was NOT Sam, but someone wearing his skin, Sam tried to convince himself when he remembered the certain events that had happened in bouts of terror in the middle of the night. Soulless Sam was the fifth, or the absence of the fifth, and a definite demon in his own way, killing Sam. Except Sam couldn't die, not without his soul, or sleep, or think with emotions, and in the end Sam didn't want to die (or live properly, as it was) and almost killed Bobby because of it. Soulless Sam was indeed the aftermath of Lucifer and Lucifer's Cage, but hopefully the effects of it would wear off soon.
But then again, how would Sam ever heal from himself?