[perrygeo]

I wrote a data ingest process, a simple web crawler that parsed and processed data from html pages and inserted to a database. This is a side project for me - no resources to maintain a k8s cluster or batch processing system or orchestration layer. Just a good old unix process running on a cheapo VPS. The process would periodically wake up and get to work, sleep, repeat.

I used Python with polars for the first iteration. It worked but I was plagued by downtime, constantly hitting runtime exceptions that I hadn't considered. I rewrote it in Rust (also using polars) and it wasn't necessarily faster but it was rock solid. Every code path was explicitly handled and it would loop forever; there was literally no way to cause a panic. And it's been running for over 2 years without a single bug.

Rust is the language if you have a well-defined data crunching problem to automate. It's not so great for exploratory work but for production code that needs to be rock solid, consider it. Being able to sleep at night without babysitting your data pipeline is nice!

[jochem9]

This x10.

When working with high variety data there are so many things that can go wrong (esp when working with external data sources), so you want a strongly typed compiled and memory safe language.

I really believe that as data engineers we need to adopt a language like Rust. It helps a lot that it has polars, so we can keep working with the dataframe paradigm.