It may be technically illegal, but when I worked for the 2010 Census we were told that it's fine if someone doesn't want to answer a question, and that's what I told people when I interviewed them.
According to this article, no census failures have been prosecuted since 1970.