[wakawaka28]

Not to spam you, but this is probably a function that would not work for AD: https://math.stackexchange.com/questions/2383397/differentia...

That is, it is a series that converges, but trying to take the derivative as a sum of individual terms results in divergence. I learned a lot of this type stuff ages ago, but in 2025 I just searched for an example lol... I am long overdue for a review of numerical analysis and real analysis.

ChatGPT also says something about an example related to some Fourier series, maybe related to this: https://en.m.wikipedia.org/wiki/Convergence_of_Fourier_serie... You can ask it all about this stuff. It seems pretty decent, although I have not gone too far into it.

[adgjlsfhk1]

Don't worry, This is interesting! AD should work on this example (at all points where the derivative converges) see this desmos graph for a very informal proof that the series converges https://www.desmos.com/calculator/djf8qtilok.

The place where I think we're talking past each other here is that in infinite precision, AD perfectly differentiates your algorithm, but even an algorithm using arbitrary (or even infinite) precision math, that to high accuracy controls the error of a differentialable problem, AD can still do weird things.

[wakawaka28]

Try that with `g(x, i) = sin(ix) / i`. I think that is one that ChatGPT said wouldn't work, as in you can't get the derivative of `f(x)` term-by-term. I guess another issue that could happen is that the original sequence converges, and the derivative sequence converges, but they converge at different rates. So code that calculates the function to sufficient precision would not automatically get the derivative to any particular error threshold.

[adgjlsfhk1]

> g(x, i) = sin(ix) / i

That's an example where the derivative does not exist.

> I guess another issue that could happen is that the original sequence converges, and the derivative sequence converges, but they converge at different rates.

This is a lot closer to what's happening in the video. For a potentially simpler example than an ODE solver, if you had a series evaluator that given a series evaluated it at a point, AD would need a similar fix to make sure the convergence test is including the convergence of the derivative.