跳到正文
DEV Community· Isaac Land·· 3 小時前AI 評分59

EEG 分類器可能因辨認受試者而取得高分

EEG classifiers can score well by recognising the person

AI 導讀

EEG 分類器若在訓練集和測試集包含同一受試者的數據，可能靠辨認受試者取得偏高評分，令結果高估對新受試者的泛化表現。

正文

Many EEG machine-learning papers report accuracies that look too good. One common reason has nothing to do with the brain signal you care about: the model has learned to recognise who a recording came from. If the same person's data sit in both the training and the test set, the test score answers an easier question than the one the paper claims to answer. This post walks through how that happens, what a recent small replication found, and how to set up the split so your number means what you think it means.

How windowing creates the problem

A typical EEG pipeline cuts each recording into many short windows (say, 4 seconds each), extracts features per window, and treats each window as a sample. A study with 36 participants can easily end up with a couple of thousand windows. That looks like plenty of data.

The trap is the next step. If you shuffle all the windows and do a random 80/20 split, windows from the same person land on both sides. Consecutive windows from one recording are highly similar, and so are different recordings from the same head. A flexible classifier can then learn person-specific boundaries ("for this participant, the alpha power shifts like this between rest and task") that do not carry over to anyone new.

The randomisation unit decides what you are estimating:

  • Window-level split: how well the model classifies new windows from people it has probably already seen.
  • Subject-level split: how well the model transfers to people it has never seen.

Neither is wrong in itself. The problem is reporting the first and claiming the second.

What a minimal replication showed

In October 2026, Renish Joshi published a deliberately simple replication in Towards Data Science. He used the public EEGMAT dataset on PhysioNet (EEG during mental arithmetic): 36 healthy students, each with a resting recording and a serial-subtraction recording, sampled at 500 Hz with 10/20 electrode placement (Zyma et al., 2019).

The pipeline was kept fixed:

  • 19 scalp channels, 1–40 Hz band-pass filter, non-overlapping 4-second windows (2,134 windows in total).
  • Welch power spectra, relative power in theta, alpha, beta and low-gamma bands, giving 76 features per window.
  • Three classifiers: class-weighted logistic regression, an RBF-kernel SVM, and a 300-tree class-weighted random forest. Scaling sat inside a scikit-learn pipeline, so it was fit on training data only.

Only the evaluation changed. The window-level protocol used 10 stratified random 80/20 splits. The subject-level protocol used 6-fold GroupKFold with participant ID as the group, so no test participant contributed a single training window.

Balanced accuracy dropped for every model:

Classifier Random windows Held-out subjects
Logistic regression 0.770 0.648
RBF SVM 0.864 0.602
Random forest 0.664 0.532

ROC-AUC moved the same way. Because a linear model, a kernel method and a tree ensemble all showed the pattern, it is hard to blame one model family.

Do the features know who you are?

A score drop alone doesn't explain the mechanism, so Joshi ran a second test: use the same 76 spectral features to predict which of the 36 participants a window came from. Chance is about 2.8%. The identity classifier reached 95.6%. On a class-balanced subset (15 rest and 15 task windows per person, 1,080 windows), it still reached 94.0%, with 99.5% top-3 accuracy.

He is careful about what this shows. It does not prove the task classifier used identity as its decision rule. It shows the features carry a lot of person-specific information, which is enough to make overlapping splits optimistic. The window-versus-subject gap also persisted on the balanced subset, so class imbalance alone doesn't explain it.

The averages also hide a wide spread. Per-participant balanced accuracy for the subject-level logistic regression ranged from 0.289 to 0.956, and 5 of 36 participants scored below chance.

This isn't a one-off

The replication follows Brookshire et al. (2024) in Frontiers in Neuroscience, who reran two published convolutional networks. For Alzheimer's disease versus subjective cognitive decline, test accuracy was 99.8% with segment-level splits and 53.0% with subject-level splits, where the confidence interval included chance. For seizure detection on the Siena Scalp EEG database, accuracy fell from 79.1% to 65.1%. In a non-exhaustive review of 63 deep-learning EEG papers on disease identification, they found only 17 (27.0%) that clearly avoided this leakage.

Setting up the split correctly

The rule is simple to state: match the split to the generalisation claim. If you claim the model works on new people, no data from a test person may reach training, and that includes anything learned from data along the way.

In scikit-learn, that means grouped cross-validation, with every learned step inside the pipeline:

from sklearn.model_selection import GroupKFold, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

# X: (n_windows, n_features), y: labels, groups: participant ID per window
pipe = make_pipeline(
    StandardScaler(),
    LogisticRegression(class_weight="balanced", max_iter=1000),
)

cv = GroupKFold(n_splits=6)
scores = cross_val_score(pipe, X, y, groups=groups, cv=cv,
                         scoring="balanced_accuracy")
print(scores.mean(), scores.std())

A few things to check beyond the outer split:

  • Tuning: if you tune hyperparameters, do it with an inner grouped split inside the training participants only (nested CV), not on the outer test folds.
  • Feature selection, PCA, ICA, normalisation: anything fit on data belongs inside the pipeline so it only sees training participants.
  • Report per-subject results: show the spread across held-out people, not just the mean.
  • Run the identity check: if your features can name the participant, window-level results are likely to be inflated.

Window-level evaluation is still fine when the question really is "new sessions from people already enrolled", for example a personalised model that only ever serves the users it was trained on. Just say so.

Limits worth keeping in mind

Joshi's study is one 36-person dataset with hand-crafted spectral features. The two protocols also differ in resampling structure (10 random splits versus 6 grouped folds), so the gap is descriptive, not a formal causal estimate of leakage. Where the person-specific signal comes from (physiology, electrode placement, session effects or residual artefacts) isn't resolved. And as of mid-October 2026, the article page did not link a public code repository, so the numbers can't yet be rerun independently. Brookshire et al. did publish code in their supplementary material, and EEGMAT is freely downloadable.

I first wrote up this case in Chinese, with the figures from Brookshire et al., on Scholay's research-methods section. The same logic applies to any data with repeated samples per person: patients, users, sensors, or schools.

References

來源：DEV Community · dev.to

#教學/實踐#數據/訓練