Alas, many financial models do well in backtests and then fail in the real world. You have to expose them to all kinds of market conditions and not just the recent one. Good luck out there!
Yes even before AI, backtesting was a crapshoot. But AI adds another crease because it might have knowledge in its training. If you’re training an LLM model on backtesting, it might know that Apple crushed a certain quarter and knowing that, it buys shares before that Q earnings.