股票排序模型應勝過的兩項隨機對照測試
Random referees: two controls every ranking model should beat
Quantooo Lab 以兩項隨機對照檢查股票排序模型是否勝過機遇，以及訓練流程有否資料洩漏。一項是隨機挑選股票或打亂日期區塊，另一項則以隨機打亂的標籤訓練完整模型；LightGBM 的打亂標籤版本得分約為零。作者的模型結果及測試的十二個模型系列，均未能明確勝過隨機對照。
A model that ranks stocks with an information coefficient (IC) of 0.02 sounds like something. Is it? The quickest way to find out is to compare it with things that can't have skill.
We run two kinds of random referee next to every result.
1. Shuffled rankings (is the result better than chance?)
Keep everything about the strategy the same (number of stocks, rebalancing dates, costs), but pick names at random. Do it many times. You get a distribution of results that pure luck produces with your exact setup.
Two flavours are useful:
- Cross-sectional shuffle: each day, shuffle scores across stocks. Tests whether the ranking itself carries information.
- Time-block shuffle: shuffle whole blocks of dates. Keeps the time structure, so it catches results that come from one lucky period.
If your model sits inside the cloud of random results, it hasn't shown skill, whatever the headline number.
2. Shuffled labels (does the pipeline leak?)
Train the full model, with the same features, the same walk-forward schedule, the same everything, on labels that have been randomly shuffled. There is nothing real to learn. The IC should come out around zero.
If it doesn't, something in your pipeline is leaking the answer: a feature computed with future data, an overlapping window, a join on the wrong date.
For our LightGBM model, the shuffled-label version scored about zero, which is what you want to see.
What we found
Our score and the twelve model families we tried didn't clearly beat the random referees. Seeing that side by side is more useful than any single number: it shows how big "nothing" looks in this data.
Try it
Before you trust any ranking model, ask two questions: how does it compare with random picks, and what does it score on shuffled labels? If nobody ran those, the result isn't finished.
From Quantooo Lab, the makers of quantooo.com, a free, open and educational quant lab: every formula, every test and every failure is public. See it here: https://quantooo.com/research
Educational, not investment advice.
來源：DEV Community · dev.to