我以 6,000 組模擬數據集測試回測驗證工具，結果它只「説謊」一次
I tested my backtest validator on 6,000 fake datasets. It lied once.
作者以 6,000 組模擬數據集測試自建回測驗證工具，3,000 組無優勢數據中僅 1 組被判定為全部通過。扣除每筆交易 0.2% 額外成本後，每筆交易真實優勢為 0.15% 和 0.10% 的數據集仍分別有 15.6% 和 3.6% 通過，顯示無優勢噪音測試的低誤判率不適用於成本高於收益的策略。
Note: I used an AI assistant (Claude) to help write this article and the code. The numbers come from my own locked simulation run, not live trading. This is not financial advice.
I built a small Python tool that checks a list of backtest trades against eight strict criteria and prints PASS or FAIL. Before I show it to anyone, I wanted to know one thing: how often does it say PASS when there is nothing there?
So I locked a protocol first (hypothesis, scenarios, and the rule for what counts as acceptable), then ran it once. No tuning after seeing the numbers.
The setup
Each fake dataset is a series of dates with 2 trades per date, return = true mean + noise. I made 500 datasets per scenario, 12 scenarios, 6,000 datasets in total.
Six scenarios have no edge at all:
- N1 to N3: plain Gaussian noise with 200, 100 and 400 dates
- N4: fat-tailed noise
- N5: noise plus 3 planted huge winning dates
- N6: an edge in the first half of the data that reverses in the second half
Six scenarios have a real edge of 0.05%, 0.10%, 0.15%, 0.20%, 0.30% and 0.50% per trade, with 200 dates. The tool also subtracts 0.2% per trade as extra cost, by design.
Result 1: false passes on no-edge data
Out of 3,000 no-edge datasets, the tool said "all criteria passed" once. It happened in N1: 1 of 500, or 0.2% (95% interval 0.0% to 1.1%). The other five scenarios gave 0 of 500, including the planted outliers and the half-and-half edge. My rule was that up to 1% false passes is acceptable. The upper end of that interval is slightly above 1%, so one honest reading is "probably under 1%, not proven".
Which criteria do the work? In N1 the strict t-statistic (at least 3) passed in only 0.2% of datasets, and "still positive after removing the 3 best dates" passed in 29.6%. The simple checks (first half positive, second half positive, median positive) pass about half the time on pure noise, which is why none of them is enough alone.
Result 2: how big must a real edge be?
Share of datasets with a true edge that passed all criteria (200 dates, noise sd 1% per trade):
- 0.05% per trade: 0.4%
- 0.10%: 3.6%
- 0.15%: 15.6%
- 0.20%: 50.4%
- 0.30%: 98.2%
- 0.50%: 100%
The tool is conservative about small edges, but it is not safe below the cost line. After the extra 0.2% cost, an edge of 0.15% per trade is a net loss, yet the tool passed 15.6% of those datasets, and 3.6% at 0.10%. Those are false passes on strategies that lose money after cost, well above my 1% rule. So the 1% false-pass result only applies to pure noise, not to weak real edges that cost more than they earn. At 200 dates a real edge needs to be about 0.2% to 0.3% per trade before it is detected reliably. That is a limit of this tool, and I plan to examine it before I release anything.
What this does not show
This is a simulation with independent days. Real trades have regimes and clustering, so real false-pass rates can be higher. I ran the tool at its default settings, and the 500-dataset runs use fewer permutations (2,000) than the real tool (10,000).
I can post the locked protocol and the raw output in a comment if anyone wants to check the numbers.
I am keeping a waiting list for the tool at https://botgauntlet.com. Which of these checks do you already run on your own backtests?
來源：DEV Community · dev.to