I completely disagree. Backtesting matters, especially in HFT. What would you rather have: an algorithm that has performed well on 5 years worth of historical data or one that has performed well in the past 2 weeks.
Neither; these are techniques applied on observed data in an obviously dynamic open domain. The data is uncontrolled - so we have no idea if it is representative of the current state of the domain theory - does it cover the distributions properly; we don't know. We know the domain is dynamic in that the world economy moves like around alot and we can't predict these movements, in the sense that most actors did not call the last substantial negative market movement. The domain is open in that new features can appear that drive the value of the variable of interest; for example the availability of property in China, new battery technology, a patent on nuclear fusion.
We do not have techniques that account properly for any of these things.
You may as well draw lines on charts and sell that advice, you'd probably get as good a hearing in any Investment Bank in the world !
He's not wrong, backtesting really is not the same as testing on real live data. You are fitting your models on data that has already been done, without any of the risks factored in. Even if you do find a system that picks every ups and downs in the backtest, it will do poorly with live trades because the volatility and risk is something that changes based on fundamental data.
However, in this case, the fact that no backtest or any sort of risk/reward ratios were published suggests high level of skepticism.
In my experience? The later. Regime change is a killer in trading algos.
Further, accurate back testing is very difficult for lots of reasons, the most obvious being your own actions aren't reflected.
Most algo firms use historical data to develop a hypothesis & then many different ways to validate that hypothesis. Test trading being the one with the most weight.
We do not have techniques that account properly for any of these things.
You may as well draw lines on charts and sell that advice, you'd probably get as good a hearing in any Investment Bank in the world !
oh....
wait...