Hacker News new | ask | show | jobs
by chrisjc 2306 days ago
I'm unfamiliar with BQ, but from what I understand BQ doesn't have any concept of streams... with maybe an exception for change-streams (change tracking) and that's very different and less powerful than what Materialize is able to do.

After all, in BQ you're only able to query what has been already has been ingested or is being digested and that is somewhat by definition not 'real-time'.
1 comments
manigandham 2306 days ago
GCP has PubSub which is their streaming/messagebus product. Cloud Dataflow is their stream/batch processing engine (backed by Apache Beam).

There's a new feature called "Dataflow SQL" that lets you submit a declarative SQL query through the BQ UI as a job to Dataflow which can join both BigQuery tables and PubSub topics in the same job.

https://cloud.google.com/dataflow/docs/guides/sql/dataflow-s...

link
chrisjc 2305 days ago
Thank you for the correction. Sounds like a very slick solution bringing batch and streaming (the B and EAM obviously) together.
link