[mcbrown]

Former professional investment manager here...

The biggest problem with things like this, which almost nobody talks about in the context of investing, is publication bias.

100 people try to develop a profitable trading algorithm. 1 comes up with one that looks great on back-tests at a 1% confidence (in other words, exactly what you'd expect from random chance alone over 100 trials).

That person writes an article/pitch/business plan based on their algorithm. You never see results from the 99 who failed.

Going forward, the successful algorithm is no more likely to work than the failed 99, but from the perspective of the general public it sure looks like a winner!

[JoeAltmaier]

There's an old con game - you send 500 letters to gamblers, predicting the next Dodgers game. 250 predict they'll win; 250 say lose. Game happens, 250 people think Hey lucky guess. To those you send 250 letters, 125 predict they'll win the next game; 125 lose. After 6 games you have 8 people who have seen you guess right 6 times in a row. Get them to pay you for another (worthless) letter.

[branchless]

So much this. Also even if you have won it means very little going forward. If you put 100 guys in a room and asked them to try to flip N consecutive tails one guy will come out thinking he is the king of flipping, with a rock-solid "system". He's just someone who doesn't understand probability. And as you say you don't hear from the other 99 including the math guy who flipped N/2 heads and is muttering about it.

[mathgenius]

> 100 people try to develop a profitable trading algorithm....

It's much worse than this with machine learning approaches. Imagine a million people trying to find a profitable algo, all on your laptop, and you are choosing the best one out of all of those.

If you are used to pen-and-paper trading strategies, or even excel spreadsheets, machine learning is just a completely different level to this. And probably how it works will be unintelligible to anyone. I don't even see how someone can write a business plan based on this.

[selectron]

The type of approach used has limited effect on survivorship bias, what matters is the number of people employing different approaches and the size of the effect. So if machine learning approaches can produce real results, the data will show this. Survivorship bias is real, but it is not the full story.