Roughly, the number of upvotes on their stories and comments minus the number of downvotes. These don't match up exactly; some votes are dropped to prevent abuse.
Awesome, back to 51 points. Is it because I'm talking about karma and get automatically downvoted? I don't usually mention karma and yet still my little 1 point doesn't get added to my "score."
Usually, I write a short reply on some thread and although my post shows me "1 point by anthay" my karma doesn't change. I'm sure it used to.