[haraldurt]

The first plot of "Activation 0" appears to actually be the random input, if it corresponds to hs[0] in the code. The rest of the activation plots seem to all be strictly non-negative. The other plots with negative values are of gradients, not activations.

[rvarma]

Ah yeah, my bad, I should've instead shown the activations after the first layer, since "activation 0" is just the distribution of the random data I started with