Hacker News new | ask | show | jobs
by fractionalhare 1916 days ago
I can see a lot of potential in Go for data engineering specifically, yeah. Those would probably be some very stable and performant ETLs. And the concurrency and network primitives would make it easy to develop libraries like Prefect/Airflow.
2 comments
MrPowers 1916 days ago
Yep, agreed. Go is a great language for AWS Lambda type workflows.

Python isn't as great (Python Lambda Layers built on Macs don't always work). AWS Data Wrangler (https://github.com/awslabs/aws-data-wrangler) provides pre-built layers, which is a work around, but something that's as portable as Go would be the best solution.

link
fractionalhare 1915 days ago
Love awswrangler. I use that over boto whenever I have the opportunity.
link
RSHEPP 1915 days ago
We use Go for our ETL, with some Python too. We are in the process of transitioning to Argo Workflows from a K8s CronJob/Job setup which has been pretty stable itself.
link