JavaScript Tooling for Data Pipelines

[floyergilmour]

JS fullstack devs out there, what tooling are you using for creating data pipelines or for orchestrating data jobs? do you have to wander into SQL/Python land for this? or do you hack something together your self? As someone new to the JS community I have struggled to find a tool or platform for this. In Python land you have tings like Airflow and Dagster for example

[cotera-grant]

We're a "data team as a service" type company and we actually just open sourced our TypeScript library for doing data pipelines. We're still pretty early in the open sourcing / docs process but we've used it in production for over a year now and are really happy with the stuff we've built on top of it.

A big bonus for us as fellow full stack JS devs is that the full compiler and type checker and SQL gen runs in the browser which we use to build cool stuff

https://getnasty.dev/ https://news.ycombinator.com/item?id=39641713

[floyergilmour]

Super cool!!! Is it almost like Drizzle but for BQ/Snowflake?

I don't understand from the website if NASTY also is meant to just run typescript code to do (lets say webscraping, update some postgres DB, check for missing billing stripe billing etc etc)