Counterpoint: I have a HomePod mini in one of my rooms and it reacts easily to any mention of Siri in other rooms. So I am actually impressed with how good it is in reacting.
I had a similar issue where a HomePod mini in our living room always responded when in the kitchen, so I just disabled the microphone on the living room HomePod minis.