Vern never really considered his dad family. Sure he called him dad, went through the motions, but he wouldn't call him family.
His sons were family, but they were both dead now. His daughter-in-law was dead. His granddaughter might as well be, he'd probably never see her again.
His wife was dead.
Waking up in the morning Vern would go through who was left that he considered family and the list was painfully short some days. The days that he remembered that Robson had been family and Vern had as good as signed his death warrant by exiling him from the brotherhood.
The days he hesitantly acknowledges his part in Andy's death.
The closest thing he had to family now was the brotherhood and even he knew how sad that was. That the closest people to him, the ones he cared the most about or cared about at all anymore really, were a bunch of criminals.
Which isn't so bad, except that realizing it always brings with it the realization that the only people in the world that care about him are criminals.
And as much as Vern is proud of his life and makes the best of it, he still knows the truth. Good people don't only associate with criminals. And it should have already been clear to Vern that he wasn't a good person, but somehow until he lost everyone it had escaped him and those few and far mornings apart he takes the extra moment to acknowledge that its too late to change it now.