甚麼是過度擬合：AI 模型怎能在訓練數據上得分 99%，卻在真實環境中失效？

Imagine building an AI model to detect fraudulent transactions.

You train it on thousands of historical transactions. The results look incredible. It correctly classifies 99% of the training examples.

You deploy it, expecting it to catch fraud reliably.

But something goes wrong.

Legitimate transactions get flagged. New fraud patterns slip through. The model that looked almost perfect during training performs disappointingly on transactions it has never encountered before.

How can a model that performed so well suddenly become unreliable?

The problem might not be the model's architecture, its computing power, or even the quality of the training algorithm.

It might have learned the wrong things.

This is called overfitting, one of the fundamental problems in machine learning. It occurs when a model learns patterns that work extremely well on its training data but fail to generalize to new, unseen examples.

Understanding overfitting means understanding a critical distinction: learning the training data is not the same as learning the underlying problem.

1. The Difference Between Learning and Memorizing

Consider a student preparing for an examination.

One student memorizes the answers to every question in a practice paper. When the exact questions appear in the examination, the student performs exceptionally well.

Another student understands the underlying concepts. Even when the examination introduces unfamiliar questions, the student can reason through them.

Both students may achieve the same score on the practice paper. But only one has demonstrated that their knowledge transfers to new situations.

Machine learning models can behave in a similar way.

During training, a model receives examples and adjusts its internal parameters to reduce prediction errors. The goal is not simply to reproduce the training examples. It is to learn patterns that remain useful when new data arrives.

An overfitted model can become excessively specialized in its training examples. It may learn meaningful relationships, but it also learns accidental details, noise, and coincidences that do not hold outside the dataset.

For example, imagine training an image classifier to distinguish wolves from dogs.

Suppose most wolf images in the training dataset contain snow, while most dog images show grass or houses.

A model might discover that snowy backgrounds are strongly associated with wolves.

Its training accuracy could be excellent. But it has learned a shortcut rather than a reliable way to distinguish the animals.

When shown a wolf standing on grass, or a dog playing in snow, it may make the wrong prediction.

The model has learned a correlation in its training data, but that correlation is not a dependable solution to the actual task.

This is the central danger of overfitting: a model can learn patterns that explain its training data without learning patterns that generalize.

2. Why Does Overfitting Happen?

Overfitting is not caused by a single problem. Several factors can contribute to it.

An Overly Complex Model

A model with too much capacity relative to the available data may be able to fit not only meaningful patterns but also random fluctuations in the training examples.

Imagine fitting a curve to a small collection of data points.

A simple model might capture the general trend while missing some individual points. A much more flexible model might pass through every point, including those that are there because of measurement errors or random variation.

The second model fits the observed data more closely, but that does not necessarily make it better at predicting new observations.

The same principle applies to neural networks, decision trees, and other machine learning algorithms.

However, complexity alone does not guarantee overfitting. The outcome also depends on the amount and quality of data, the model's inductive biases, the training procedure, and the regularization techniques used.

Too Little or Unrepresentative Training Data

A model can only learn from the examples it receives.

Suppose a medical image classifier is trained using images collected from a single hospital. The images may share particular scanners, acquisition settings, patient demographics, or image-processing conventions.

The model may learn patterns associated with those conditions.

When deployed at another hospital with different equipment or patient populations, its performance may decline.

The problem is not necessarily that the model memorized individual images. It may have learned relationships that were valid in the original dataset but are not sufficiently representative of the broader population.

Small datasets make it easier for accidental patterns to appear convincing. Unrepresentative datasets create a different but related problem: the model may never have learned the variation it will encounter in production.

Training for Too Long

Machine learning models often improve as training progresses, but the improvement does not necessarily continue indefinitely.

Early in training, a model may learn broad, useful patterns. As training continues, it may begin fitting increasingly specific details of the training data.

At some point, its performance on unseen data may stop improving even while its training performance continues to get better.

This is one reason training duration matters.

However, overfitting is not inevitable in every long training run. Its likelihood depends on the model, dataset, optimizer, regularization, and other training conditions.

Noisy Labels and Accidental Patterns

Training data is rarely perfect.

Human annotators make mistakes. Sensors introduce measurement errors. Datasets contain duplicates, inconsistent labels, and unusual examples.

A sufficiently flexible model can learn some of these errors instead of treating them as noise.

Imagine a dataset containing photographs of cats and dogs, but a few images are incorrectly labeled. A model that tries to fit every example may learn to reproduce those incorrect labels.

The same issue appears in financial forecasting, medical diagnosis, fraud detection, and natural language processing.

When the training data contains unreliable signals, a model can spend capacity fitting details that have little value in the real world.

3. Training Accuracy Can Be Misleading

If training accuracy does not tell us whether a model generalizes, what should we measure?

The first step is to evaluate the model on data that was not used to fit its parameters.

Machine learning workflows commonly divide data into three sets.

Training set: The model uses this data to learn its parameters.

Validation set: Developers use this data to compare model configurations, tune hyperparameters, select checkpoints, and make other training decisions.

Test set: This data is reserved for a final evaluation of the selected model.

These sets serve different purposes.

If a model achieves 99% accuracy on its training set but only 78% on a representative validation set, the gap is a warning sign. The model performs much better on the examples it learned from than on unseen examples.

This is often called a generalization gap.

But there is an important qualification: a gap alone does not prove overfitting. Training and validation sets may differ in difficulty, class balance, or data distribution. Evaluation metrics may also hide problems affecting particular groups or classes.

The split must be designed carefully.

For example, randomly splitting medical images can produce misleading results if images from the same patient appear in both training and validation sets. The model might benefit from patient-specific characteristics instead of learning patterns that generalize to new patients.

In that situation, splitting by patient is more appropriate.

Similarly, when predicting future events, randomly splitting observations across time can allow information from the future to influence training. A chronological split is often more suitable.

A validation set is useful only when it represents the kind of unseen data the model will encounter.

4. The Hidden Danger: Data Leakage

Sometimes, a model appears to generalize well because the evaluation process accidentally gives it access to information it should not have.

This is known as data leakage.

Consider a model designed to predict whether a customer will cancel a subscription.

The dataset includes customer activity, account details, and a field indicating whether the customer has already completed the cancellation process.

If that field is accidentally included as an input, the model may achieve excellent evaluation results.

But the feature reveals the answer rather than helping the model predict it.

A less obvious example occurs during preprocessing.

Suppose developers normalize a dataset using statistics calculated from the entire dataset before splitting it into training and test sets. Information from the test set has now influenced the preprocessing applied to the training data.

Depending on the transformation and task, this can introduce leakage and make evaluation less reliable.

The safer approach is to split the data first, fit preprocessing steps using only the training set, and then apply the learned transformations to validation and test data.

Leakage can also happen through duplicated records, overlapping users, future information, or features created after the event the model is supposed to predict.

It is important to distinguish leakage from ordinary overfitting. Overfitting occurs when the model fits training-specific patterns too closely. Leakage occurs when information improperly crosses the boundary between training and evaluation, or when an input contains information that would not legitimately be available at prediction time.

The two can coexist, and both can produce models that fail after deployment.

5. How Do We Reduce Overfitting?

There is no single technique that eliminates overfitting in every situation. Effective solutions depend on why the model is overfitting.

Collect More Representative Data

More data can help the model distinguish recurring patterns from random variation.

But the quality and diversity of the data matter as much as its quantity.

Adding thousands of nearly identical examples may provide less value than collecting examples from new environments, devices, users, or conditions that the model must handle.

For the wolf-versus-dog classifier, the dataset should include wolves and dogs in a variety of backgrounds, lighting conditions, poses, and environments.

The objective is to expose the model to the relevant variation instead of allowing a single accidental feature to dominate.

Use Regularization

Regularization introduces constraints or penalties that discourage certain forms of complexity in a model.

For example, L1 and L2 regularization penalize large or numerous parameter values in different ways. These techniques can discourage a model from relying too heavily on particular parameters.

Decision trees can be regularized by limiting their depth or requiring a minimum number of examples in a leaf.

Neural networks can use techniques such as dropout, which randomly disables selected activations during training to discourage excessive dependence on particular units.

These methods work differently, but they share a broad objective: encourage solutions that are less likely to depend on fragile, training-specific details.

Regularization is not a substitute for representative data or proper evaluation. A heavily regularized model can still perform poorly if the data is biased or the task is poorly defined.

Use Data Augmentation

Data augmentation creates modified versions of training examples while attempting to preserve their relevant meaning.

For image classification, this might involve flipping images, cropping them, or changing brightness within reasonable limits.

For speech recognition, it may involve controlled changes to audio. For text, augmentation requires more care because a seemingly minor change can alter the meaning or correctness of a sentence.

The goal is to make the model less dependent on incidental details.

For example, showing an animal classifier the same species against different backgrounds can help discourage it from using background appearance as the primary signal.

Augmentation must be appropriate for the task. A transformation that changes the label or removes important information can make training worse.

Use Early Stopping

Early stopping monitors performance on validation data while training progresses.

If validation performance stops improving, developers can stop training and restore the checkpoint that performed best on the validation set.

The idea is simple: don't continue optimizing the training objective indefinitely if doing so no longer improves performance on unseen data.

In practice, developers often use a patience threshold because validation metrics can fluctuate from one checkpoint to another.

Early stopping does not guarantee the best possible model, but it can be a useful and computationally inexpensive defense against overfitting.

Simplify the Model When Necessary

A model that is too flexible for the available data may benefit from a simpler architecture or stronger constraints.

For a decision tree, reducing its depth may prevent it from creating branches that fit individual training examples.

For other models, developers may reduce the number of parameters, limit feature complexity, or choose an architecture with more appropriate inductive biases.

However, making a model simpler is not automatically better. An overly simple model may fail to capture meaningful patterns, resulting in underfitting.

The objective is not to build the smallest model possible. It is to find a model that captures useful structure without depending excessively on accidental details.

6. The Opposite Problem: Underfitting

Overfitting is only one side of the problem.

A model can also be too simple to learn the underlying patterns in the data. This is called underfitting.

Imagine trying to predict a complex relationship using a model that can represent only a straight-line trend. If the true relationship is much more complicated, the model may perform poorly on both training and validation data.

A useful diagnostic is to examine both training and validation performance.

• Low training error and substantially higher validation error: This can indicate overfitting.
• High training error and high validation error: This can indicate underfitting, although poor data quality or optimization problems can also cause it.
• Low training and validation error: This suggests the model is fitting the training data and generalizing well on the validation set, though further evaluation may still be necessary.

These patterns are diagnostic clues, not absolute rules.

The challenge is to balance model flexibility with generalization. A model must be powerful enough to capture meaningful patterns, but not so dependent on its training examples that it fails when conditions change.

7. Why Passing a Test Set Still Doesn't Guarantee Success

Suppose a model performs well on its training, validation, and test sets.

Is it ready for production?

Possibly, but there is another question: how closely do those datasets resemble the real environment?

A randomly sampled test set from historical data may not represent future users, new devices, changing customer behavior, or evolving fraud techniques.

This problem is especially important when the data distribution changes over time. A model trained on yesterday's patterns may struggle when tomorrow's environment behaves differently.

This is often discussed under the broader concept of distribution shift.

For example, a fraud detection model may learn patterns associated with previously observed fraud campaigns. Once deployed, attackers can change their behavior, making some learned patterns less useful.

Even a model that generalizes well to a held-out test set can become less effective as the world changes.

That is why production systems need more than a one-time accuracy score.

Depending on the application, developers may monitor prediction quality, false positives, false negatives, calibration, data drift, and performance across relevant groups.

They should also establish procedures for investigating degradation and retraining when appropriate.

In high-stakes applications, evaluation must reflect the actual consequences of errors. A medical screening system, for instance, cannot be judged adequately by overall accuracy alone if false negatives and false positives have very different costs.

A model is not finished when training ends. Its reliability must be assessed in the context where it will be used.

The Bigger Picture

Overfitting exposes a fundamental challenge in machine learning.

A model is trained on a finite collection of examples, but it is expected to make predictions about cases it has never seen.

Training accuracy tells us how well the model fits those examples. It does not, by itself, tell us whether the model has learned patterns that will remain useful beyond them.

That is why representative datasets, careful validation, leakage prevention, regularization, and production monitoring matter so much.

A model that achieves 99% accuracy may be excellent. Or it may have learned shortcuts, memorized accidental details, benefited from leakage, or been evaluated on data that does not reflect real-world conditions.

The score alone cannot tell us which.

The real test of a machine learning model isn't how perfectly it remembers its training data. It's how reliably it handles the data it has never seen.