Disclaimer: I do not own Charlie or Numb3rs, but after only one episode, I am already in love! Expanding on the quote "People lie, numbers don't." First Numb3rs fic here, I believe! Go me! :)
Every mathematical problem can be solved. Even if there's no solution, there can be an answer: simply, 'no solution'. Everything can be organized. And once the problem is finished Charlie can lock up his proverbial office and go home for the night.
People aren't like that, though. People are infinite variables. Most of them don't have solutions. Unsolvable people are the norm, and that's what he doesn't understand.
Numbers have rules. Adding some makes bigger numbers, and taking some away makes smaller ones. They're predictable. Clean.
People aren't predictable. They aren't clean. So it's no surprise Charlie likes numbers better.