Premature optimization would be doing things in static HTML even though the rest of your code is in React because static HTML is "lighter" or "faster."
Also, Next.js does static HTML export out of the box if that's your preferred end target.
Over-engineering perhaps, but not premature optimization. You don't have to reach for that stuff until you need it.
As a side note, I do often advocate for somewhat over-engineering projects by starting with familiar frameworks. They strongly constrain the solution space yet give power to quickly meet inevitable feature creep. Strong KISS/YAGNI proponents might balk at that endorsement, but I've had far more trouble handling growing pains from adapting codebases that don't use frameworks than I have working in such over-engineered framework-based projects.
Also, Next.js does static HTML export out of the box if that's your preferred end target.