AWS has Aurora Serverless (https://aws.amazon.com/rds/aurora/serverless/) for that purpose with MySQL and Postgres compatible engines. But I haven't heard anyone using it for side projects yet.
Note that you can either run Aurora Serverless constantly (at a cost of about $43 a month) or have ~30 second startup times if the instance has timed out (~15 minutes).
They also have RDS Proxy (https://aws.amazon.com/rds/proxy/) that lets you pool connections from tons of lambda instances in order not to overwhelm your DB when scaling up.