[liuliu]

Yeah, luckily, you can unit tests these and fix them. They are not concurrency bugs (again, luckily).

BTW, numeric differentiation can only be tested very limitedly (due to algorithmic complexity when you doing big matrix). It is much easier / effective to test against multiple implementations.

[antoine-levitt]

You can easily test a gradient using only the forward pass by doing f(x+h) ~ f(x) + dot(g, h) for a random h