[hombre_fatal]

You're just describing any software problem where things aren't static or you haven't yet mapped the domain. Storing the whole response in the database so that you aren't losing data is a good solution while you make your logic robust against the domain of values.

You run into this talking to an undocumented API for example. But notice that to "query it later", you still have to write/polish the rest of your logic to handle the actual data which is how you end up with a system that doesn't need to store every API response to disk. This isn't a web thing, it's any time you don't have a 100% handle on the interface between two systems.

Nothing to do with Rails, though. You were trying to create harder to change solutions based on a premature understanding of the values you were working with which caused churn in solutions (like DB columns) that are annoying to change frequently.

But in that situation, you as an engineer have to be able to step back and go "okay, what can I do to handle this at a more dynamic layer until I understand what I'm working with?"