That's what I do. If a call comes into my phone from a number that isn't in my address book, it just gets sent straight to voicemail. My phone won't even ring.
The only reason I don't just drop the call completely is to cover situations where a someone not in my address book might be calling me for something important.