Kobo has a bookstore that’s pretty comprehensive - I haven’t found anything missing. Not sure that gets you out of DRM land, but at least you’re not giving money to Jeff Bezos.
For public domain books, I use Standard Ebooks, Project Gutenberg, and Internet Archive, generally in that order.
For copyrighted books, anywhere as long as it provides DRM‐free EPUB or PDF.
• Humble Bundle introduces a nice sale every few days. Key marker for DRM‐free: “Use on Any Device”. Representative recent purchases: complete Peanuts (42 vols.) for $25, complete Wheel of Time (17 vols.) for $18, complete Malazan (17 vols.) for $18, complete Lone Wolf and Cub (28 vols.) for $18… I check Humble pretty regularly now.
• Google Play. Key marker for DRM‐free: “Content protection: This content is DRM free.”
• Barnes and Noble. Key marker for DRM‐free: “At the Publisher's request, this title is being sold without Digital Rights Management Software (DRM) applied.”
Amazon rolled out DRM‐free ebooks (for some books) earlier this year, but at this point they’ve permanently lost my business.
In this case the reason for dropping support is most likely that the only DRM they can support on that older hardware has been broken. There's no technical reason why it can't be supported, and I doubt it would cost them much (or even anything) to continue support.
Meanwhile, I can still read physical books I've had since I was a child, 40 years ago. The Kindle is undeniably more convenient than physical books, but this is absolutely an unnecessary sunset of these devices.
My Kindle 4 hardware works great, I still read it nightly. Since it doesn’t feel like it’s obsolete (in fact it has physical buttons so may be slightly better than a modern Kindle), it feels like a blatant cash grab by Amazon to get us to buy new devices that probably are laden with ads or other revenue generators.
I've stuck with Kindle, but that's 80% inertia (Amazon has most books, the device works well enough) and 20% existing library is Kindle e-books.