[mdda]

One of the things about financial markets is that large numbers of people are attempting to spot patterns, and eek out a profit from the patterns repeating. People are very good at this - which means that, over time, the number of profitable patterns reduces. Thus, to human eyes, market behaviour becomes noise : just like zip compression reduces a bytestream to being essentially white noise by taking out repetitive sequences.

Computers are just the next step, crunching out the patterns until they are unexploitable (below the threshold of trading costs).

The end result is that markets are a random walk - unless you are at the bleeding edge with faster machines, better latency, lower transaction costs, etc.

Of course, an alternative to this is to do true bottom-up analysis, or invest in illiquid companies (like VCs do).

[marchdown]

This is a strong point, and the one rarely acknowledged at that. Many people stop at pointing out that efficient markets cannot be gamed, and almost all exploitable inefficiencies have been ironed out already. In correctly observing that most investment activities are fueled by greed and human biases, they incorrectly extend this to trading illiquid goods, which still have a lot of low-hanging fruit to pick.

[radikalus]

One counterpoint is that at the bleeding edge of low-latency, pattern analysis and sig-int become once again extremely meaningful.

Can you execute your strategy faster than your opponents if you go to a slightly more aggressive (less arbitrage-y) signal?

Should you? (Why bother if you're faster?)

For what situations is it worth "thinking longer"? Some straight arbs require speed beyond what you can do if you want to use your HOT "smart" model.

If you work outwards from the fastest "stupidest" trades, there's a vast array of strategies/opportunities that intersect ML/AI, hardware design, network optimization, and so forth -- I do agree that if you're looking at bad, inaccurately sampled tick data, the opportunities aren't really there anymore. (Because there's increasingly more players correcting relative value mispricings)

[disgruntledphd2]

I think the real difficulty for prediction of the stock market is that your theory will be understood by others who will alter their behaviour in response to their theory, in essence invalidating it. Its actually a problem across the social sciences, witness the inflitration and acceptance of Freud and Jung's ideas which are no so much part of the culture that theories cannot be built on them (they were always a little suspect anyway though).