Then you deal with the baggage that comes from such a label, and clarifying that you're a 'secular humanist' doesn't give you the right to expect to avoid that baggage.
1. 'baggage' is not a synonym for 'rights' (legal? social?)
2. While an Atheist is necessarily a non-believer, a non-believer is not necessarily an Atheist.
Furthermore, you tell people you're an Atheist specifically because it conveys a lot with a little. Some of what it conveys is definitely not secular humanism, if that's how you prefer being viewed, you should start with that.
Not only is that not an offensive observation, it should be common sense.