ChatGPT 無法回測：為何聲稱有 10 年回測的「AI」交易機械人並不可信
ChatGPT Can’t Backtest: Why “AI” Trading Bots With 10-Year Backtests Are Fiction
依賴即時 ChatGPT 呼叫作出交易決策的機械人，不能以模型尚未推出時的歷史數據進行回測；聲稱有完美 2019 年回測的「ChatGPT 驅動」機械人，並不能證明其 AI 交易表現。文章列出五項警號，包括缺乏公開實盤紀錄、隱藏復原系統及只展示沒有虧損月份的紀錄。驗證真正由 LLM 驅動的交易系統，應先試用至少 30 天，再投入少量資金，並記錄每筆交易及所有成本。
A bot that needs live ChatGPT calls cannot have a 2019 backtest — ChatGPT didn't exist. That single fact exposes most “AI EAs” on the market.
Here is a fact that eliminates most “AI trading bots” on the market in one sentence: a bot whose decisions require live calls to a language model cannot be backtested — because the model did not exist for most of the backtest window.
ChatGPT launched in 2022. A product showing you a flawless 2019 backtest for its “ChatGPT-powered” bot is showing you fiction — or, more charitably, a conventional strategy with AI branding. Real AI decisions require live API connectivity, cost money per call, and take seconds per decision (roughly 3–7 seconds end to end in published breakdowns). You cannot retroactively generate historical decisions from a model that did not exist, and you cannot backtest a system whose response times and model versions change underneath you. GPT-5 thinks differently from GPT-4. A backtest run on one is not evidence about the other.
The five red flags that expose fake AI EAs
Perfect long backtests. Five to ten years of flawless “AI-optimised” results. If the AI did not exist for those years, the backtest is not AI. Period.
No live record. No public wallet, no vault, no exchange-verified account. Only screenshots — which can be cropped from a cherry-picked window.
Hidden recovery systems. “Our AI intelligently scales positions for optimal recovery” is usually a martingale system doubling down on losses, described in softer words.
Annualised short windows. A good fortnight dressed as a triple-digit APR. Always ask: over what window, in what market, net of what fees?
No losing months shown. Every real trading record has losing months. A record without them is a curated exhibit, not a record.
How to validate a real AI trading system, then?
Forward testing. Since backtesting is impossible for genuine LLM-driven bots, the only honest validation is: demo for at least 30 days, then small live capital, with every trade logged and every cost counted. If a vendor refuses forward testing and offers only a backtest, that refusal is the answer.
The broader lesson applies to every trading claim, AI or not: recompute from raw data, beat the baselines, demand the trial count, check the folds. A claim that cannot survive those checks is marketing, whatever technology word is printed on it.
BuildIT Index applies that standard to any prediction file: recompute accuracy, beat baselines, check folds, count trials, return a verdict with an exit code.
If a vendor refuses forward testing and offers only a backtest, that refusal is the answer.
This article is educational, not financial advice. Past results do not predict future returns.
來源：DEV Community · dev.to