Wow, not sure how to interpret your experience that a RPi wasn't powerful enough to manage watering a few plants. I can only suspect the overall software setup is massively bloated.
If you want to run EspHome inside HA, and you recompile the devices (every release of EH), you want a decent processor/disk. The ESP stuff is a surprisingly heavy compile for a puny microcontroller.
A recent RPi is sufficient to handle a few plants - though, yes, recompilation will take time. The ESP is a beautiful piece of software, hence I highly recommend it. My native language has an expression that describes this situation perfectly: the appetite grows with eating. Next thing you know you have 2k or more entities, and your HA even handles some video feeds.
The important thing is that it's pretty much always easy to make an upgrade thanks to the good design of the backup system. Don't forget to set up backups in either case, it's a sin to not use such a complete system :)
Home Assistant is indeed a massive pile of software, mostly Python. I couldn’t get it to work reliably (or, at least, usably - the web interface was painful to use) on a Pi Zero because of memory requirements and disk access speed.
…having said that, as the other poster alludes to, it’s peak requirements that are problematic. If your device can handle them, it’s not a massive power suck because idle requirmements are low.