That's wild. I must have said "I am going back to the office" organically hundreds of times.
For example, if I went to lunch with a colleague and had to get back to the office for a meeting or to get back to my computer. Or if I was at an off-site meeting that just ended and I was going to trek back to the office.
I don't say it much anymore, as WFH has cut all this needless commuting from my life, but I use to say that constantly.