I wouldn't say it's necessarily so clear-cut that you could call it "unethical" but it definitely requires a different set of values. The classic hacker ethos involves collaboration, a disregard for artificial boundaries like national borders, and a distaste for secrecy. Working for the NSA involves lots of secrecy, working to undermine the efforts of the crypto community, and buying into some level of American exceptionalism. (For example, it's believed that they knew about Heartbleed but didn't disclose it because they wanted to use it for themselves. Even as a US citizen I'm not sure I'd be comfortable putting the national security interests of the US ahead of the interests of my friends and colleagues around the world.)
You're willingly assisting an organization with very unethical goals. It doesn't matter whether you're a cryptographer, janitor, developer, or whatever.