| eh, yes, and no. like, i see the authors point about backend data systems things like databases, query engines and storage. for the *low level* systems side of data engineering, yeah i see what you’re saying. for the higher level *take super hacky code from data scientists and turn it into a production grade pipeline* i just don’t see it right now. the thing is, right now at least, i would take the data science python code, clean it up then make it run via pyspark (or just multiprocessing). it’s still the fast happy path in my experience. this means i don’t have to rewrite the whole thing in rust from scratch, and the data science team can continue to contribute to it with a bit of hand holding (i have zero confidence in an average data scientist learning rust). also, i can run it on a single machine, or over a cluster when we start getting more data throughput. and i can set up autoscaling so i don’t have to pay attention when that’s happened. (caveat: unless there’s some super simple python->rust processing library that basically mimics pyspark that i probably don’t know about?) so, like, yeah, i see the point from the low level side of things. but at the higher level… i don’t think it’s there… yet. better title might have been “why rust is taking the low level data systems world by storm” addendum: i am actually learning rust at the moment (not for data stuff) and it has struck me as a decent language for data engineering pipeline work because of the emphasis on well defined structures (strong typing). is basically defining a concrete data model. i will check out the authors autumn of rust thingy blog post series at some point. might be helpful. |