[yevpats]

This is why we built CloudQuery (https://github.com/cloudquery/cloudquery) an open source high performance ELT framework powered by Apache Arrow (framework is open source, our connectors closed source). You can run local pipeline and write plugins (extractors) in Go, Python, Javascript and any other language and save data to any destination (files, SQLite, DuckDB, PostgreSQL, ...)

(Founder here)

[joshcanhelp]

Oh, this is really fantastic, I'm going to check it out! Does you end up with an archive of the data that's being queried?

In case we might be a good fit ... https://www.joshcanhelp.com/hire-me/