[jeeeb]

I always randomly log a sample of my inputs to TendorBoard to manually review what my training data actually looks like and (hopefully) pick up on bugs like these. Similarly I find logging high loss inputs very informative.

Coincidentally I find this article timely as I was recently reviewing PyTorch DataLoader docs regarding random number generator seeding. It’s the kind of thing unit test don’t pick up since it only occurs when you use separate worker processes.