[kuzehanka]

I've had a few Rust lovers come and mention this project to me recently. None of them had any data science or ML experience. None of them knew that Python is just used to define the high level architecture.

At the same time, comparatively tedious languages like Rust will never attract data science practitioners. They don't care about the kind of safety it brings, they don't care about improving performance in a component that's idle 99% of the time.

The bulk of the load in an DL workflow is CUDA code and sits on the GPU. Even the intermediate libraries like cublas would only see marginal-to-none benefits of being reimplemented in rust.

This is a cool project, but it has no chance to displace or even complement Python in the data science space.

[danielscrubs]

I am a data scientist and I care. The time when you could just do proof of concepts or a PowerPoint presentation is long behind us. So now we have to start to take it into production, which means we get the exact same problems as SE has always had.

Iff Rust helps us take it into production we will use it.

But it’s a lot of land to cover to reach Pythons libraries so I’m not holding my breath.

That said, Pythons performance is slow even when shuffling to Numpy.

[kuzehanka]

I must be missing something. Modern data science workloads involve fanning out data and code across dozens to hundreds of nodes.

The bottlenecks, in order, are: inter-node comms, gpu/compute, on-disk shuffling, serialisation, pipeline starvation, and finally the runtime.

Why worry about optimising the very top of the perf pyramid which will make the least difference? Why worry if you spent 1ms pushing data to numpy when that data just spent 2500ms on the wire? And why are you even pushing from python runtime to numpy instead of using arrow?

[gbrown]

Not everyone operates at that scale, and not every data science workload is DNN based

I agree with your general point, however, but the role I'd hope for with Rust is not optimizing the top level, but replacing the mountains of C++ with something safer and equally performant.

[MiroF]

But the title of this post is Python vs Rust, not C++ vs Rust. Maybe BLAS could be made safer but i don't think that's what's happening here

[likeabbas]

A big push for NN is to get them running in real time on local GPUs so we can make AI cars and other AI tech a reality. 2500ms could be life and death in many scenarios

[danielscrubs]

Every small thing counts when you have big data which is exactly why you need performance everywhere, if Rust can help with that I don’t mind switching my team to that.

The problem are usually when you do novel feature engineering not the actual model training.

But I was a C++ dev before checking the assembly for performance optimization so I guess I have more wiggle room to see when things are not up to snuff. If I got a cent for every: -you are not better than the compiler writers you can’t improve this.

Especially from the Java folks. They simply don’t want to learn shit, which is fine if they just where not so quick with the lies/excuses when proven wrong.

[kahnjw]

This is just not true. The python runtime is not the bottleneck. DL frameworks are DSLs written on top of piles of highly optimized C++ code that is executed as independently from the python runtime as possible. Optimizing the python or swapping it out for some other language is not going to buy you anything except a ton of work. We can argue about using rust to implement the lower level ops instead of c++. That might be sensible though not from a perspective of performance.

In a "serving environment" where latency actually matters there are already a plethora of solutions for running models directly from a C++ binary, no python needed.

This is a solved problem and people trying to re-invent the wheel with "optimized" implementations are going to be disappointed when they realize their solution doesn't improve anything.

[danielscrubs]

Yes. Let’s say you want certain features of a voice sample. You need to do that feature engineering every time before you send it to the model. Doesn’t it make sense to do it in C++ or Rust? This is currently already done. So if you already are starting to do parts of the feature engineering in Rust why not continue?

Yeah it’s not reasonable right now because Python has the best ecosystem. But that will not always be the case!

[kahnjw]

I can’t exactly tell what you mean but I think you’re confusing two levels of abstraction here. C++ (or rust) and python already work in harmony to make training efficient.

1. In tensorflow and similar frameworks the Python runtime is used to compose highly optimized operations to create a trainable graph. 2. C++ is used to implement those highly optimized ops. If you have some novel feature engineering and you need better throughput performance than a pure python op can give you, you’d implement the most general viable c++ (or rust) op and then use that wrapped op in python.

This is how large companies scale machine learning in general, though this applies to all ops not just feature engineering specific ones.

There is no way that Instagram is using a pure python image processing lib to prep images for their porn detection models. That would cost too much money and take way too much time. Instead they almost certainly wrap some c++ in some python and move on to more important things.

[pohl]

The python runtime is not the bottleneck.

This smells like an overgeneralization. Often things that aren’t a bottleneck in the context of the problems you’ve faced might at least be an unacceptable cost in the context of the 16.6 ms budget someone else is working within.

[kahnjw]

In what circumstance would one measure end to end time budget in training? What would that metric tell you? You don't care about latency, you care about throughput, which can be scaled nearly completely independently of the "wrapper language" for lack of a better term, in this case that's python.

It seems some commenters on this thread have not really thought through the lifecycle of a learned model and the tradeoffs existing frameworks exploit to make things fast _and_ easy to use. In training we care about throughput. Thats great because we can use a high level DSL to construct some graph that trains in a highly concurrent execution mode or on dedicated hardware. Using the high level DSL is what allows us to abstract away these details and still get good training throughput. Tradeoffs still bleed out of the abstraction (think batch size, network size, architecture etc have effect on how efficient certain hardware will be) but that is inevitable when you're moving from CPU to GPU to ASIC.

When you are done training and you want to use the model in a low latency environment you use a c/c++ binary to serve it. Latency matters there, so exploit the fact that you're no longer defining a model (no need for a fancy DSL) and just serve it from a very simple but highly optimized API.

[kuzehanka]

Novel feature engineering? Like this? https://towardsdatascience.com/python-performance-and-gpus-1...

[danielscrubs]

Looks good. I’ve tried Numba and that was extremely limited.

Current project we can’t use GPUs for production so we can only use it for development. Not my call, but operations. They have a Kubernetes cluster and a take it or leave it attitude.

We did end up using C++ for somethings and Python for most. I’d feel comfortable with C++ or Rust alone if there was a great ecosystem for DS though.

[ldng]

I see a graph on ... logarithmic scale ? No unit ? I don't know what that benchmark means.

[corndoge]

Good lord, hopefully latency isn't 2.5 seconds!

[kahnjw]

Latency in training literally does not matter. You care about throughput. In serving, where latency matters, most DL frameworks allow you to serve the model from a highly optimized C++ binary, no python needed.

The poster you are replying to is 100% correct.

[dnautics]

quote is 'data spend 2500 ms on the wire'. That's not latency. For a nice 10GbE connection, that's optimistically 3 GB or so worth of data. Do you have 3GB of training data? Then it will spend 2500 ms on the wire to distribute to all of your nodes as part of startup.

[sjwright]

I can’t even. How could you ever get 2500 msec on transit? That’s like circling the globe ten times.

[justinclift]

Maybe a bunch of SSL cert exchanges through some very low bandwidth connections? ;)

Still, it's more likely a figure used for exaggeration, for effect.

[MiroF]

You are a data scientist who seems to lack an understanding of how deep models are productionized..

I do so not unfrequently and I don't see how rust bindings would help me at all

[danielscrubs]

or maybe, just maybe, you know nothing of optimizations, so you just go: impossible!

[MiroF]

the optimization you are describing is premature - you don't need rust to productionize your models and in most cases you don't even need to be coding in a low-level language at all.

[danielscrubs]

I don’t think I’ll change your mind.

You clearly have more experience than I do that can solve everything in Python instead of C++ and/or CUDA.

You win.

[MiroF]

I mean, it'd be easier to change my mind if you had a single reason behind anything you've claimed.

I would love to understand why Rust would be more effective for productionizing ML models than the existing infrastructure written in Python/C++/CUDA.

[Nullabillity]

I'm by no means a specialized data scientist, but I've done some (very surface-level) text crunching with both the TF/Numpy stack and Rust.

To me, the nice thing about switching to Rust for that kind of stuff was that it dramatically raised the bar of what I could do before reaching for those hyper-optimized descriptive libraries.

Want to calculate the levenshtein distances of the cross product of 100k strings? Sure, just load it into a Vec, find a levenshtein library on crates.io, and it'll probably be fast enough.

Could it be done in Python? Sure, but with Rust I didn't have to think about how to do it in a viable amount of time. Does that mean that Rust is going to take over the DS world? Probably not in the short term, Rust currently can't compete with Python's DS ecosystem.

But if I'm doing something that I don't know will fit into an existing Python mold (that I know about) then I'll strongly consider using it.

[smallnamespace]

> But if I'm doing something that I don't know will fit into an existing Python mold (that I know about) then I'll strongly consider using it.

The thing is, for anything performance intensive and scientific, you're almost guaranteed to find a Python binding.

It has all these bindings because scientists are almost always writing either Python, C++, or Fortran (with a smattering of R or Octave on the side).

Want to do linear algebra? NumPy is always there, otherwise you can go lower level: https://docs.scipy.org/doc/scipy/reference/linalg.blas.html

Old-school optimization? https://www.roguewave.com/sites/rw/files/attachments/PyIMSLS...

Computer vision? https://pypi.org/project/opencv-python/

In fact, I'd basically argue against using most language-native implementations of algorithms where performance is at stake, because most implementations don't have all the algorithmic optimizations.

[danielscrubs]

And they will cover a small part of what is found in research papers.

Python has the best ecosystem, but Rust was made by a competent team so we will root for it.

[xxxpupugo]

CUDA is an important part of the story.

I think the industry is moving to 'MLIR' solution (Yes, there is a Google project called exactly that, but I am referring to the general idea here), where the network is defined and trained in one place, then the weights are exported, delegated to optimized runtime to be executed.

If such trend furthers down, then there will be very little reason to replace Python as the glue layer here. Instead it will become that everything is training in Python -> exported to shared format -> executed in optimized runtime, kind of flow.

Rust's opportunity could be to replace C++ in this case. But do mind that this is also a competitive business, where the computation is pushed further down to the hardware implementations, like TPUv1 and T4 chips etc.

[skohan]

Personally I hope for alternatives because I don't find Python that nice to work with compared to other languages. It's not awful, but I really miss having type and nullability errors detected by the compiler and reported to me with meaningful error messages.

Also there are a few weird things with the Python workflow, like dealing with Python2 vs Python3, and generally having to run everything in the context of a virtual environment, and the fact that it doesn't seem to really have a single "right way" to deal with dependency management. It's a bit like the story with imports in Javascript: those problems were never really solved, and the popular solutions are just sort of bolted on to the language.

It's amazing how many great tools there are available in Python, but it does sometimes seem like it's an under-powered tool which has been hacked to serve bigger problems than it was intended for.

[ImNotTheNSA]

> like dealing with Python2 vs Python3, and generally having to run everything in the context of a virtual environment, and the fact that it doesn't seem to really have a single "right way" to deal with dependency management.

Not to mention, this problem seems to be getting worse, not better. People are moving off of python 2.7, which was kind of the de-facto LTS release of python... leaving (currently) no LTS version of python and no clear path for the community to establish a new LTS version that the community will still support — there are so many releases with so many breaking changes in python 3 within the last few years that there is seemingly no consensus and no way to compromise.

> It's amazing how many great tools there are available in Python, but it does sometimes seem like it's an under-powered tool which has been hacked to serve bigger problems than it was intended for.

This is becoming more and more clear with every release of python IMO. The language is evolving too quickly for it to be used on large developments, but it’s still being used for that.

We have an entire test framework and libraries for high performance embedded systems level testing which is written entirely in python. The amount of high speed operations (timing tests, measurements, etc) is very obviously overwhelming for the intention of the language and libraries, yet the company keeps pushing ahead with this stuff. In order to mitigate the issue, we are developing more high speed embedded systems to offload the work from the test framework and report measurements to the framework. I think it’s quickly becoming extremely expensive and will only become more expensive — the framework is extremely “pythonic” to the point of being unreadable, using a number of hacks to break through the python interpreter. Jobs much better allocated to readable C++ libraries are being implemented in unreadable python spaghetti code with no clear architecture - just quick-and-dirty whatever-it-takes python.

I love python but I think it’s a quick-and-dirty language for a reason. What python does well cannot be beat by other languages (for example, prototyping), but I think it is often misused, since people can get something up and running quickly and cleanly in python, but it eventually has diminishing (and even negative) returns.

[llukas]

Can you name breaking changes in between python 3.x versions?

[mplanchard]

Python 3.7 made async a reserved word, which broke the widely used celery package up until their most recent release.

Raising a StopIterationErorr in a generator in 3.7 now raises instead of ending the iteration, which broke several functions at my workplace.

3.8 has several further backwards incompatible changes incoming: https://docs.python.org/3.8/whatsnew/3.8.html#changes-in-pyt...

[cwyers]

There are alternatives -- Julia and R both have bindings for Keras, which gets you Tensorflow, CNTK and Theano. (I think there are bindings for Tensorflow directly from R and Julia as well.) Once you have a trained model, it doesn't really matter from the production standpoint how you trained it.

[MiroF]

Python should and will be replaced, but not at all for any of the reasons mentioned in this thread.

A good ML language is going to need smart and static typing. I am so tired of having to run a whole network just to figure out that there's a dimension mismatch because I forgot to take a transpose somewhere - there is essentially no reason that tensor shapes can't just be inferred and these errors caught pre-runtime.

[ddragon]

Do you have an example of a tensor library that keep track of shapes and detect mismatches at compile time? I had the impression that even in static languages having tensors with the exact shape as a parameter would stress the compiler, forcing it to compile many versions of every function for every possible size combination, and the output of a function could very well have a non deterministic or multiple possible shapes (for example branching on runtime information). So they compromise and make only the dimensionality as a parameter, which would not catch your example either until the runtime bound checks.

[MiroF]

If you explain a little more of what you mean, I might be able to respond more effectively.

> I had the impression that even in static languages having tensors with the exact shape as a parameter would stress the compiler, forcing it to compile many versions of every function for every possible size combination, and the output of a function could very well have a non deterministic or multiple possible shapes (for example branching on runtime information).

I was a bit lazy in my original comment - you're right. What I really think should be implemented (and is already starting to in Pytorch and a library named NamedTensor, albeit non-statically) is essentially having "typed axes."

For instance, if I had a sequence of locations in time, I could describe the tensor as:

(3 : DistanceAxis, 32 : TimeAxis, 32 : BatchAxis).

Sure, the number of dimensions could vary and you're right that, if so, the approach implied by my first comment would have a combinatorial explosion. But if I'm contracting a TimeAxis with a BatchAxis accidentally, that can be pretty easily caught before I even have to run the code. But in normal pytorch, such a contraction would succeed - and it would succeed silently.

[ddragon]

You understood it correctly. Named dimensions is certainly a good idea even in dynamic languages as a way of documentation and runtime errors that actually make sense (instead of stuff like "expected shape (24, 128, 32) but got (32, 128, 24)"). I hope it catches on.

[MiroF]

But combined with static checking, it could be very very powerful. Definitely agree re: the power even in dynamic languages, I use namedtensor for almost all my personal development now (less so in work because of interop issues)

[vbarrielle]

There's a research language that supports compile-time checking of array dimensions: futhark [1]. It's an interesting language that compiles to CUDA or OpenCL. However it's probably not ready for production (not sure if there's even good linear algebra implementations yet). It does feature interesting optimizations to account for the possible size ranges of the arrays (the development blog is very instructive in that respect).

[1] https://futhark-lang.org/

[ddragon]

Thanks, I'll look into it.

Julia for example does have an array library that does compile-time shape inference (StaticArrays), but it cannot scale for large arrays (over 100 elements) exactly because it gets too hard for the compiler to keep track, I'm definitely curious about possible solutions.

[mruts]

Normal static typing won't help you there. You would need some sort of dependent typing. For example, look at Idris.

[pjmlp]

Rust might not be it.

But AOT/JIT compiled languages that can naturally talk to the GPGPU, without 2nd language syndrome, like Julia, Swift, Java and .NET will certainly be more attractive to data science practitioners.

I can already envision those life science guys that migrate to VB.NET when they have outgrown their Excel/VBA code, to start playing with ML.NET.

[kuzehanka]

Python already gets JIT compiled to CUDA[1] and there's an entire funded ecosystem built around python+gpgpu called RAPIDS[2] which is the future of the ML space by most indicators.

I don't see any other language even making a dent in the Python ecosystem without some kind of new killer feature that can't be quickly replicated in Python.

[1] https://numba.pydata.org [2] https://rapids.ai

[hsaliak]

You beat me to making this point.

I don't see anything in the article's python code that the numba's jit decorator cannot handle. When numba works (it's rapidly improving), it's seriously impressive.

For this particular case, you should be able to get really good performance without sacrificing readability.

Also Jax - https://github.com/google/jax

[MiroF]

Swift and static type checking and compiler analysis might be that language and feature combo.

[disgruntledphd2]

Thank you for the cite to rapids.ai, that looks extremely interesting :)

[MiroF]

Your last paragraph is my nightmare

[wokwokwok]

> comparatively tedious languages like Rust will never attract data science practitioners.

Well, fast.ai is using swift now.

... I think it's fair to say 'never say never'.

You're probably right, rust isn't really the sweet spot for this stuff, but its also a case that python has some down sides that are pretty severe, and well acknowledged.

[chillee]

Just because fast.ai has some investment in Swift does not mean that S4TF has attracted mind share. The bulk of fast.ai is still taught on Pytorch.

[wokwokwok]

The parent comment literally said data science practitioners don't care about speed or safety because the GPU is where all the real work happens; that's false, I've provided an example of it being false from a respected party. What do you want me to say?

eh, I give up. Believe whatever you want to believe.

[chillee]

I'm not saying that no data science practitioners care about speed and safety, and its easy for me to come up with cases where they should/do care.

My point is that even fast.ai still views S4TF as somewhat niche, and that data science practitioners as a whole still don't care.

[MiroF]

I am a practitioner, I care, but everyone on this thread seems to be distracted by the red herring of speed and efficiency.

None of these languages are going to be speedier in the GPU. But python has serious design problems that manifest itself when developing a large project.

You can only go so far with dynamic typing

[melling]

I wouldn’t call Swift a tedious language.

With type inference, immutability, etc, Swift is far from tedious:

http://www.h4labs.com/dev/ios/swift_cookbook.html?topic=stri...

http://www.h4labs.com/dev/ios/swift_cookbook.html?topic=dict...

It’s not quite as nice as Python but it’s an enjoyable language.

[wokwokwok]

> They don't care about the kind of safety it brings, they don't care about improving performance in a component that's idle 99% of the time.

https://www.fast.ai/2019/03/06/fastai-swift/:

> Because Swift for TensorFlow is the first serious effort I’ve seen to incorporate differentiable programming deep in to the heart of a widely used language that is designed from the ground up for performance.

> But Python is not designed to be fast, and it is not designed to be safe. Instead, it is designed to be easy, and flexible. To work around the performance problems of using “pure Python” code, we instead have to use libraries written in other languages (generally C and C++), like numpy, PyTorch, and TensorFlow, which provide Python wrappers. To work around the problem of a lack of type safety, recent versions of Python have added type annotations that optionally allow the programmer to specify the types used in a program. However, Python’s type system is not capable of expressing many types and type relationships, does not do any automated typing, and can not reliably check all types at compile time. Therefore, using types in Python requires a lot of extra code, but falls far short of the level of type safety that other languages can provide.

ie. My point: OP's point is categorically false.

...not that swift is tedious. I <3 swift.

[kuzehanka]

...But fast.ai is a Python library. Partially written in swift.

Validating the original point that nothing will replace python for DL applications any time soon but middleware will continue to be implemented in c++/rust/swift/whatever you fancy.

S4TF isn't the first and certainly not the last end to end non-python DL stack. It might be worth highlighting as an example if it ever reaches mindshare above the noise floor amongst those stacks.

[wokwokwok]

> Our hope is that we’ll be able to use Swift to write every layer of the deep learning stack, from the highest level network abstractions all the way down to the lowest level RNN cell implementation. There would be many benefits to doing this...

Well, the tldr: you’re wrong.

The more approachable reading: python isn’t going anywhere, but people are looking at other things for more than just low level implementations with a python wrapper.

...it’s early days yet, who knows where things will go... but maybe do a bit more reading and have an open mind?

There’s more to life than python.

[skohan]

Python is easy to get started with, but once a project grows to any meaningful size, I would rather have a compiler which is giving me more correctness guarantees than Python is capable of.

IMO Swift strikes the best balance between strictness and productivity of any language I've worked with.

[ImNotTheNSA]

> Python is easy to get started with, but once a project grows to any meaningful size, I would rather have a compiler

boy do I have a continent full of python developers to introduce you to... “python - by any means necessary” is their motto.

But in all seriousness, there are a lot of people (where I work) who started with python and have been daily driving it for so long that any other language is too tedious to get started with. Even if it means writing unreadable, hacky python, python still wins out for them.

I suspect there are a lot of similar people in data science.

[frutiger]

Rust has those too, GGGP still considered it tedious.

[antisemiotic]

I think the tedious part would be the borrow checker. Also, Rust doesn't have top-level type inference (by design).

[skohan]

Yeah there is some tedium in general with Rust syntax. Semicolons, for example, feel old fashioned, and there's a lot of verbosity in things like unwrapping. It's a fine language and I like working with it, but there's a lot of details involved in Rust development which don't make sense for data scientists to worry about.

[skohan]

This is why I think Swift is a much better choice to replace or at least compliment Python than Rust. It has a modern, powerful type system and all the quality of life advantages which come with it, but it manages this with a lot more usability than Rust.

A well written swift framework almost becomes a DSL for the problem domain, which is a great property for a data science tool to have.

[echelon]

Swift is not fun on non-Mac platforms. It feels a lot like Google's Dart in terms of how it was positioned and advocated.

Modern Rust isn't difficult to use. This is becoming a really tired meme from detractors. The compiler is incredibly helpful, non-lexical lifetimes are a thing, and unless you're doing a lot of sharing and parallelism, you can avoid many borrow checker problems until you learn RAII.

[skohan]

It's a fair criticism of Swift that it is not well supported across platforms. The story is getting better, but it is still miles behind Rust in this regard. I am a huge fan of Swift as a language, but I am very critical of the tooling.

> Modern Rust isn't difficult to use. This is becoming a really tired meme from detractors.

I beg to differ. I've been programming professionally for over a decade, and I have shipped projects in a variety of languages, and I can safely say that Rust has a steeper learning curve and requires more cognitive overhead to use than many other languages. I find it relatively nice to work with rust in spite of this because the tooling is so great, but it's undeniable that Rust has made tradeoffs which sacrifice ease of use in favor of safety and performance.

[MiroF]

The tooling for Swift 4 TF is not anywhere near satisfactory. I can't even seem to get it on my computer without installing XCode (and that's on an apple computer)

I should be able to pull a docker image and have s4tf immediately at my fingertips

[kognate]

Seems the Fast.ai forums have got you covered here. https://forums.fast.ai/t/swift-for-tensorflow-using-gpu-with... describes how to do it, while not completely 'at my fingertips' it's pretty close. and Googles repo at https://github.com/google/swift-jupyter has a clean setup, too.

[MiroF]

I've come across that first link before, but remember I had some issues.

The two problems are a. I don't have a GPU locally b. I would rather have a compiler than just a REPL/jupyter

[lreichold]

Not local, but you can start using S4TF immediately via Google Colab.

https://colab.research.google.com/github/tensorflow/swift/bl...

[MiroF]

if I’m going to be developing, I need a compiler, not a REPL or a notebook. Haven’t managed to find that

[blub]

The "Rust is difficult" criticism isn't getting tired, but it must be getting tiresome for the Rust community because they don't have a good answer to it.

The top level comment from an early adopter in the last Rust release thread on HN was complaining about the tedious complexity.

[echelon]

I think this project has immediate merits when it comes to productionizing small, sparse networks for running on the CPU.

I do research and prototyping in Python, but I have to deploy on mobile devices. I was going to roll my own implementation, but now that this exists, it's something I'm going to look into.

[MiroF]

Load the model from torch script? In C++?

Don't see why Rust made this possible.

[echelon]

Ideally we'd have native Rust loading of tensorflow, pytorch, etc. models. I've hand-coded a few, but it was pretty monotonous.