Hacker News new | ask | show | jobs
by WalterBright 3360 days ago
Algorithms that predict the past accurately but are no better than random at the future are a dime a dozen, and I don't see how your suggested algorithm is different.
2 comments

Corollary: having a time machine that can go a few minutes back in time is worth much less than one that can go a few minutes forward in time.
This implies the time machine can go only one way. Taking information from the future back in time is worth more than taking information from the current to the future. We do the latter all the time, we do not need a time machine for this, just patience.
Then look at backtesting. The evaluation data set is out of time, meaning the better than random performance is on unseen future data. The algorithm was implemented, not merely a suggestion.
You can still leak information from your backtested time series by choosing WHICH algorithm to use out of a large pool of algorithms. You'll get regression to the mean because you optimized for a noisy signal (backtesting performance) of future earnings.
It is possible to leak information, but then you are doing it wrong. Don't use only a single out of time test set to do parameter or model selection, keep an out of time holdout set.

But really, this is the bare basic of forecasting. It is somewhat annoying to have to regurgitate all of this: Like non-leaking forecasting is impossible somehow. It would be a better discussion if everyone just assumes proper forecasting practices. Instead people seem to assume I have no clue what I am doing, discarding my technique, because I did not mention removing duplicates, scaling, proper validation techniques, ... and a 100 other things, which are of no importance to the technique itself.