[mehrdadn]

On a parallel note, search for "thresholdout". It's another (genius, I think) way to "stretch" how far your data goes in training a model. I won't do a better job trying to explain it than those who already have, so I won't try—here's a nice link explaining it instead: http://andyljones.tumblr.com/post/127547085623/holdout-reuse

[claytonjy]

I got really excited about thresholdout a couple weeks ago, but I've since cooled; setting the threshold seems like too much black magic.

I thought the Zillow blogpost [1] was a nice intro (and I'm a sucker for Seinfeld references), and it demonstrates the sensitivity-to-threshold value in a way the original academic authors never did.

[1]: https://www.zillow.com/data-science/double-dip-holdout-set/