[mritchie712]

+1

we're using Perspective in crabwalk[0] (it's like dbt specifically built for duckdb and written in rust) and it's amazing paired with duckdb. Near instant loads for hundreds of thousands of rows and you can keep everything in arrow.

0 - https://github.com/definite-app/crabwalk

[0cf8612b2e1e]

Where are you using/advocating crabwalk?

It does look interesting, but for the local ETL use case, I am missing the pitch on just having my own collection of SQL scripts. Presumably the all-local case needs less complexity. Unless the idea is that this will eventually support more connectors/backends and work as a full dbt replacement?

[mritchie712]

A few features:

* Built-in column level lineage (i.e. dump in 20 .sql files and crabwalk automatically figures out lineage)

* Visualize the lineage

* Clean handling of input / output (e.g. simply specify @config output and you can export results to parquet, csv, etc.)

* Tests are not yet implemented, but crabwalk will have built-in support for tests (e.g. uniqueness, joins, etc.)

we're using it in our product (https://www.definite.app/), but only for lineage right now.