And how do you manage the versioning and deployment of your views and mviews? What tooling do you use to refresh those materialized views on a schedule?
Is it something you've written and maintain yourself? What testing do you have on them?
I find that dbt answers all of these questions pretty simply in a way that avoids the need to reinvent the wheel. It's not some godlike piece of software, just a helpful framework wrapper around plain old SQL
> And how do you manage the versioning and deployment of your views and mviews?
Underlying SQL lives in git, along with the code. Deployment happens via normal CICD pipelines.
> What tooling do you use to refresh those materialized views on a schedule?
A database that automatically rebuilds views when the underlying table changes (ClickHouse), or if I’m using postgres, a “dumb” job that simply updates the views. Periodically.
> Is it something you've written and maintain yourself? What testing do you have on them?
Written and maintain myself? Not really, it’s either Spark or Rust code that’s a very thin wrapper to drive the underlying SQL. Everything runs in Kubernetes, as either plain deployments (for ongoing/streaming workloads) or cronjobs for periodic workloads. SQL and wrapper code are tested using the normal in-language testing tools.
> I find that dbt answers all of these questions pretty simply in a way that avoids the need to reinvent the wheel.
I don’t think people not using dbt are reinventing the wheel, in my case I’m not doing anything that wouldn’t be considered exceedingly boring in architecture and usage.
Avoiding weird proprietary tools also means that the technology and approach used is almost identical to what the devs use, which means more shared knowledge and better reusability.
I write my SQL queries in Jupyter Lab, mixed in Python code and f-strings, which
is enough of a template engine to my needs atm. I can also directly visualize stuff with pandas (etc.)
species: str = 'acer'
structure: str = 'trees.table'
query = f"""
SELECT * FROM {structure} WHERE species = {species}
"""