Hacker News new | ask | show | jobs
by nunie123 1927 days ago
I agree with much of what you said.

However, I'm disinclined to believe there will be a trend away from a centralized data warehouse (or data lake).

There is inherent value in having a single source of truth for analytics. With modern cloud tools abstracting away the complexity distributed analytical databases, data warehousing is getting easier and more powerful.

It's true that there is added complexity in centralizing the data. But as the author of this article suggested, you're in a bad spot if your marketing team and your sales team can't agree on last month's revenue. I'm not sure how you'd solve that problem in an architecture where the data isn't getting centralized.
1 comments
michael_j_ward 1927 days ago
> There is inherent value in having a single source of truth for analytics.

> I'm not sure how you'd solve that problem in an architecture where the data isn't getting centralized.

Isn't the entire point of this article that the author achieved a single source of truth without moving the data by using postgres's Foreign Data Wrappers?

link
nunie123 1926 days ago
The data is being moved, it's just that the mechanism to move it was the FDWs, instead of e.g. an Airflow DAG. The FDWs are interesting because they're update-on-query as opposed to updating on a fixed schedule or update on a source system event, which is more common.
link
michael_j_ward 1925 days ago
> The data is being moved

My understanding is that FDW's act as a proxy. I don't believe the tables are materialized.

Can you link me where it says otherwise? I don't see anything in the docs [1].

[1] https://www.postgresql.org/docs/current/postgres-fdw.html

link