Unfortunately bigger phone means more space for the battery and rest of the hardware, so companies will keep making their Pro Ultra models larger and so of course the larger ones are more of a status symbol.
But the screen consumes most of the charge in a modern phone anyway. The smaller the screen, the less battery capacity you need to get a sensible battery life. The CPU also matters, of course — that's one thing I like about my Pixel 4a, it lasts long even after 3 years because it uses a weak-ass SoC that nonetheless is perfectly capable for messaging and web browsing.
The screen is off most of the time. The iPhone 13 Mini has the same CPU as the larger models and there is a significant difference in battery life, you likely won't find a review for this device which doesn't mention how fast the battery drains.
From my own experience, iPhones are really bad with idle battery life. The iPhone 11 Pro I have for occasionally testing iOS apps would drain the battery in ~3 days just laying on my desk doing nothing. Android phones, on the other hand, last more than a week when not used thanks to "doze mode" and other optimizations in recent versions. But I suppose that difference isn't as noticeable when you actually use an iPhone as your real phone and charge it daily at night.