[worik]

This jumped out at me: "The obvious next step to take was to move to SQL"

No. Not unless your data is relational. This is a common problem, relational databases have a lot of over head. They are worth it when dealing with relational data. Not so much with non relational data.

[bradfitz]

Maybe "obvious" was the wrong word there. What we meant to say was that would be the typical route people would go, doing the common, well-known, risk-averse thing.

[worik]

Yes, I agree.

It is the wrong "obvious" thing. Sounds like they did better though