To me it's the difference between a few line exchange that is briefly distracting vs switching to working on a completely different task, getting up to speed on it just in time to have to switch back and refamiliarize myself with where I left off. Order of minutes vs hours.
In my experience the questions tend to cluster towards a small group of individuals; so from their perspective, you're the 5th interruption in line for the day and they've barely been able to start their own work.
That's one reason why I like the idea of taking forced breaks; they give you a scheduled time to handle brief interruptions, and if you're fairly regular people can tune in to your schedule(e.g. if you're always "working" the first 50 mins of an hour and "resting" the last 10 mins, it's easy to know whether you'd be able to quickly answer a question). Of course, the forced breaks are another form of an interruption from flow; it's still a trade-off.