[Mz]

"Hey, your opinion in stand up this morning was awesome, can you sit in on this meeting."

There's your problem. Stop outing yourself. Don't give your opinion on things you don't want to be dragged into. Zip your lip. Then you probably don't have to say "no" to anything because they probably won't ask. Give opinions on those things that are related to the domain you desire to work in and then make polite noises about "not my area" or something if pressed on these other things that you don't really want to work on.