Trello don't allow any more fine grained access than this. Power-ups can request view and write access and that is about it. No way to say we only want access to the current board or etc. Or that we only want to be able to update labels/members but not add comments.
They have been talking about changing that, but I understand it is complex and never really becomes a priority
It does seem that the trade-off generally between handing a 3rd party access to content vs. improved functionality provided by plugins is rarely worth it. This not only applies to Trello but many other work/productivity tooling where the content is generally sensitive - eg. Slack, Trello, Gmail etc.
I'd be curious to know how many people work at companies that allow 3rd party plug-ins for these kinds of tools.
They have been talking about changing that, but I understand it is complex and never really becomes a priority