[teekert]

I think I’m experiencing this where I work. The Data Lake is quickly gaining traction and feature requests poor in: please incorporate FHIR genomics resources, please make a UI for this image type, place make import filters to extract meta data from these files… this team seems swamped now. The solution would be to give more power to the requesters? Allow them to access underlying technologies, implement their own data models? Seems logical. Am I understanding this correctly?

[siganakis]

Yes, you are understanding it correctly. The idea is that you give the "requesters" access to the data, then enable them to do their thing with it (with training / support / shadowing) and publish their results as "data-products" so that others can leverage it too in their own "data products".

The "data mesh" is essentially the collection of these independent "data-products".

We already see management problems with self-service analytics like PowerBI, Tableau & Looker. Its too easy for people to create dashboards / reports that are subtly wrong and which cause confusion. There is a balance between empowering to build data products and centralised control. Too much empowerment of people who don't understand the right way to do something leads to a horrible mess of contradictory data. Not enough, and people can't effectively do their job. Governance and process is the key to finding the balance and enforcing it.

The issue with the data-mesh is that there isn't really any great tooling to support the management or development of data products, or a data-mesh generally. I am sure this will change over time as vendors start building hype around it.

[apahwa]

a bit self serving but I would recommend reading about Airbnb's Minerva (which I created). we leverage this data mesh concept to allow teams to define data independently and then Minerva handles blending the data from different teams together with guaranteed consistency.

you can read more here: https://medium.com/airbnb-engineering/airbnb-metric-computat...