They can get a warrant to pick your lock and install a keystroke logger, or bust in while you're using your computer, forcibly separate you from it, and download the encryption key from memory.
Right, I'm not arguing that. My point is that it's considerably more difficult for them to go this route; they need to convince a judge to affix his signature (and his credibility) to a warrant by demonstrating the existence of probable cause.