[vsskanth]

How does this compare to zygote ?

[GiggleLiu]

1. Scalar level

Various benchmarks (including those in the paper) show NiLang is much better than Zygote to differentiate scalar functions. And Zygote is much faster than TF and PyTorch.

2. Tensor level

Zygote, TF and PyTorch are much better than NiLang, because NiLang's matrix multiplication is not fully optimized, it is much slower than BLAS. (One can wrap BLAS into NiLang, but that does not measure NiLang's programming language level AD performance anymore)

[cbkeller]

As far as I understand it (which is not very far):

Zygote calculates derivatives using source-to-source automatic differentiation.

This calculates function inverses (so to stretch the analogy a bit, it's kinda like "source-to-source automatic inversion")

[ChrisRackauckas]

Not necessarily: you can use it to build fast inverse of scalar functions. Here's an example on a vjp of an ODE f:

https://gist.github.com/GiggleLiu/0c4608f70bda050f59992f5fc0...

[cbkeller]

I'm probably just missing something, but did you mean to say "fast derivatives of scalar functions" by any chance? I.e., that you can use the free inverses to get at the derivatives in some way?

[ChrisRackauckas]

oh yes, I meant derivatives.