Bugger is not common, but also not uncommon American slang. It has rather lost it's original meaning. I have heard little old American ladies use it, probably having heard it in a movie, with the intention of a meaning derived from to bug (bother). I don't think any slang dictionary has ever caught up with this usage.
I've used bugger since the first of the Austin Powers movies. My friends and I used it until one of them married a South African girl and she made us stop being as it was incredibly impolite and rude apparently. I started again last year after watching Top Gear and then recently got yelled at by a new accounting manager who moved into the office across from my cubicle. Once again, I was being crude but in all honesty, for me, the word is no different than saying frag, frell, crude bunnies, etc.
Very commonly used in New Zealand, its used on TV with no problems, heres the iconic Toyota ad using the word bugger to good effect: http://www.youtube.com/watch?v=TKY_OysWu3k