[ogogmad]

I don't know whether this belongs here, but...

Formally, there is a generalisation of differentiation which can handle functions like ReLU (i.e. locally Lipschitz non-differentiable functions) by allowing a derivative to be set-valued. It's called the Clarke gradient. The Clarke gradient of ReLU at 0 is the closed interval [0,1]. Note that the Clarke gradient doesn't satisfy the chain rule (except in a weakened form) which might seriously mess up some assumptions about autodiff. Is this generalised derivative useful in autodiff?

I imagine that this is a largely theoretical tool that's useful in analysing algorithms but useless for actually computing things.

[medo-bear]

i havent heard of clarke gradient before. convex analysis has something called subgradient [0], is it different?

[0] https://en.m.wikipedia.org/wiki/Subderivative

[ogogmad]

The subgradient in convex analysis is a special case of the Clarke gradient. The subgradient is precisely the Clarke gradient for convex functions. Convex functions are always locally Lipschitz except in weird cases.

[edit]

Question: Are there numerical applications in which the subgradient is actually computed, or is it a purely analytical tool?

[agnosticmantis]

(Stochastic) subgradient methods are used in practice to optimize non-differentiable convex functions. They have a slower convergence rate than (stochastic) gradient descent though.

See for example : https://www.stat.cmu.edu/~ryantibs/convexopt-F15/lectures/07...

[fault1]

yes, see: https://juliadiff.org/ChainRulesCore.jl/dev/maths/nondiff_po...

and related neat usages in set based optimization methods in MathOptInterface (part of JuMP.jl): https://matbesancon.xyz/post/2020-12-24-chains_sets2/

[ogogmad]

To me, it looks like you're computing a single element from the set, instead of the whole set. Which makes sense, but it's an important caveat.