Hacker News new | ask | show | jobs
by user1241320 2524 days ago
Is this somehow similar to the new Swift Automatic Differentiation feature? https://github.com/tensorflow/swift/blob/master/docs/Automat...
3 comments

Yes, it's the same idea. But Julia's differentiable programming capabilities are far more advanced and mature than Swift's. As far as I'm aware, Swift still doesn't support differentiating code with control flow (branches or loops), which, needless to say, eliminates pretty much all non-trivial programs.

Compare that to the situation in Julia: ∂P works today on arbitrary programs—like the ray tracer and other examples in this paper—programs which are highly non-trivial and use iteration, recursion, mutation and global state. All of which Julia's ∂P can take derivatives through.

When you additionally consider Swift's essentially non-existent computational and data science ecosystem, it's a bit hard (for me at least) to rationalize the Swift ∂P effort. (Are we going to differentiate iPhone apps?) They're attempting to bootstrap their computational/data ecosystem by allowing calling Python code, but as soon as you call into Python, you lose all ability to take derivatives which only works for pure Swift code. So any program which relies on Python to do some of the computation you want to take a derivative of won't be differentiable, which kind of defeats the point of having ∂P in the first place. We'll see how it pans out but the Swift effort has considerable technical and social challenges to overcome.

How do if conditions handled? They are not differentiatable right? So are loops?
There's a whole paper explaining how it all works :)
Does Swift have an equivalent to Python‘s numpy?

It seems to lack a nice way of doing vector and matrix operations.

They built in Python interop so that you can use numpy in TFSwift
Bridges always have trolls under them.

If you don't see the troll, that doesn't mean it isn't there, it's just waiting for you to cross.

Doesn’t that introduce serious overhead compared to first class native arrays as in Julia or FORTRAN?
My impression is that this is to bootstrap adoption, there is a Swift-native TensorFlow package being developed also.
Bridging into NumPy isn’t going to be useful unless it covers some AD system too (XLA?).
XLA is a compiler for array code. It doesn't come with AD -- you need a wrapper like TF or JAX (or Swift, I guess) for that.
I was imagining the FFI emitting XLA code needed to implement the TF/JAX routines behind the scenes, but it's wild speculation.
Correct: you can call Python from Swift OR you can use AD but not both at the same time.
Thanks for linking this. I have been curious how auto diff gets implemented.