You over estimate the 'semi-pro' market for graphics cards. Gamers are barely willing to pay for 20GB. There's no market for consumer cards with an order of magnitude more RAM until games are built to use that memory.
I would personally love that project but there are already so many versioning issues in the space it would be a nightmare if ROCm randomly broke things all the time.
Assuming 50 input tokens per second, you could still be waiting ten minutes for a full 32k token prompt.
What you are talking about is highly optimized inference using accelerators, batching and speculative decoding to achieve high throughout. Once you have that then compute is irrelevant except in terms of cost, but if all you have is a small consumer grade GPU you will be compute limited at the extreme limits of your context window.
But "basement ML" is a thing, the market of people who are interested in PC gaming but not to the point of being lifestyle gamers who throw every cent they can spare at that altar. The GPU they bought long before the pandemic is still running every game they throw at it, but they never completely stop eyeing the new stuff. Dipping their toes in ML, even if it's just getting through 80% of some stable diffusion setup tutorial, can be a very welcome excuse to upgrade their gaming. A card sold for gaming but with generously overprovisioned VRAM (ideally in the range of the lowest bin of the biggest or second-biggest chip I think) could match that market segment very well - and it would not only compete with other price points, it would actually increase the market by some buyers (those who would not upgrade without the "ML excuse").