Definitely something I've worried about. But a major architectural change is different than an improvement. I've left a lot of low hanging fruit for me to fix up after release. But with this change it would literally become a new, backwards-incompatible product. That's where I personally draw the line for when to wait.
Of course, I assume OP is smart enough to realize they have free will and that I am not being prescriptive! OP had stated they were planning to release it, but held off, so I believe their intent is to release it.
By the way, one can release a side project without it turning into a product. :-)
That's actually all it was in the beginning. Just writing code for fun. But the more code I wrote, the more it evolved into an app that demands to be released, especially since the nature of it is to be a platform, SDK, and API for others.