Hacker News new | ask | show | jobs
by Danylon 3362 days ago
Then look at backtesting. The evaluation data set is out of time, meaning the better than random performance is on unseen future data. The algorithm was implemented, not merely a suggestion.
1 comments

You can still leak information from your backtested time series by choosing WHICH algorithm to use out of a large pool of algorithms. You'll get regression to the mean because you optimized for a noisy signal (backtesting performance) of future earnings.
It is possible to leak information, but then you are doing it wrong. Don't use only a single out of time test set to do parameter or model selection, keep an out of time holdout set.

But really, this is the bare basic of forecasting. It is somewhat annoying to have to regurgitate all of this: Like non-leaking forecasting is impossible somehow. It would be a better discussion if everyone just assumes proper forecasting practices. Instead people seem to assume I have no clue what I am doing, discarding my technique, because I did not mention removing duplicates, scaling, proper validation techniques, ... and a 100 other things, which are of no importance to the technique itself.