Not great, agreed, but they did at least architect the OS so that they are decoupled and you can build without them very easily. In fact the largest custom ROM doesn't bundle with them by default (LineageOS). What do you need Google services for?
You can still have a useful experience on a pure AOSP build.
You'll miss features like push notifications, Google accounts outside of the browser, an app store and various other features.
But there's no reason the open source community couldn't add these, a package repo/app store equivalent and push messaging would probably be the deal breakers for me as a phone where chat apps don't work isn't useful to me.
The binary blobs to get driver support are also a problem, but it's also true in desktop linux, see for example Nvidia's graphics card driver.
I think it's better to come up with replacements for those Google services and apps instead of rewriting the entire operating system. You need to solve the same problems on a linux smartphone as well, so Android gave you a lot of time savings, and I doubt the app sandboxing is even remotely as good on those non-Android linuxes. For Google Play for example there is F-droid, for push messages/FCM there is UnifiedPush, for Google Maps there is Osmand.