They can't, but only part of the code is public domain. If you want to use the whole pile of it, you do so under the GPL - if you pick out the changes made by the govt, then you can apply those elsewhere.
Insofar as the non-GPL part is specific to the GPL code, yes. If you reimplement the GPL code under a different license, you could reuse the public domain bits. I expect in most cases, it wouldn't be terribly worthwhile.