Hacker News new | ask | show | jobs
by jas8425 1184 days ago
> how to turn data into a commodity

This is an interesting problem both for public datasets like the ones you're showing, and for internal datasets created and exposed by teams within an org. There are a lot of moving pieces to consider over and above the basics of getting data into and out of systems:

* How do you communicate the data schema in a way that provides both strong guarantees (data you see WILL match the advertised schema), while still being adaptable to change and unexpected circumstances (schemas WILL change)

* How do you deal with transformations/data cleanup in a non-hacky way? Then how do you scale them?

* How do you deal with data ownership? What if one data product consumes another in a nontrivial way -- who owns what?

I'm working on a team building a product to solve these problems! We recently opened beta signups so if you're interested, check us out: https://www.estuary.dev/

I'm happy to answer any questions :)