Our data-heavy API returns JSON responses from our DB in ~40ms using ActiveRecord. So no, it doesn't waste 100ms on every request! We're serving around 6000 requests like that per second.
Rails on Heroku is about the most inefficient combination you can get for a web server.
I've always thought of it as the platform you use for running toy apps with maybe a couple of dozen users at most and a low load. It's incredibly quick and easy to get a Rails app running on Heroku, a couple of hours at most. But you pay (in dollars and performance) for this.
Yeah, Heroku performance on shared dynos is really not great. The variability is very high also, so you'll get some requests seemingly randomly timing out. The dedicated performance dynos are better, but even more expensive. We were on Heroku for a long time but as the scale got too much we migrated to Google Kubernetes Engine which has been great.
The Heroku dynos I've tested with some years ago performed quite horribly. Only a proper Xeon server was able to achieve sub-100ms responses.
But hey, if Ruby improved in the meantime, cool.