機器學習系統能否自我修復？構建自我修復機器學習系統：第一階段

Designing an automated framework for detecting machine learning failures, diagnosing their causes, and evaluating recovery strategies.
Series: Building a Self-Healing Machine Learning System | Part 1
Introduction: What Happens When Machine Learning Models Fail?
Machine learning models are often evaluated under carefully controlled conditions. We train models, measure performance, optimize hyperparameters, and deploy the best-performing version.
But what happens after deployment?
Real-world data rarely remains unchanged.
Customer behavior evolves. Data pipelines introduce unexpected missing values. Feature distributions shift. Previously reliable predictions can gradually become less accurate.
A model that performs well during development may behave very differently when exposed to changing production data.
Traditional machine learning workflows often focus on detecting performance degradation through monitoring dashboards, periodic evaluations, and manual retraining.
But I wanted to explore a different question:
Can a machine learning system automatically detect a failure, identify its likely cause, apply an appropriate recovery strategy, and determine whether the recovered model is safe to use?
That question became the foundation of my graduate capstone research project:
Self-Healing Machine Learning System for Automated Failure Detection and Model Recovery in Credit-Default Prediction.
In this first article, I will walk through Phase 1 of the project, where I defined the research problem, selected the dataset, designed the system architecture, established evaluation criteria, and developed a reproducible experimental plan.
Rather than immediately implementing a complex machine learning model, I began by defining what failure, recovery, and successful healing should actually mean.

1. What Is Self-Healing Machine Learning? Self-healing machine learning is an approach to building ML systems that can respond to certain failures through automated monitoring, diagnosis, and corrective actions. The goal is not to create a model that never fails. Instead, the goal is to design a system that can recognize defined failure conditions and respond in a controlled, measurable, and reproducible manner. For this project, I organized the concept into five primary components:
2. Data Monitoring Continuously or periodically examine incoming data for abnormal behavior, including missing values, unexpected categories, and distribution changes.
3. Failure Detection Identify whether observed data conditions exceed predefined thresholds.
4. Failure Diagnosis Determine which monitored feature or condition is responsible for the detected problem.
5. Automated Recovery Apply a correction strategy appropriate to the diagnosed failure, such as imputing missing values or initiating model retraining when justified.
6. Recovery Evaluation Measure the performance of the recovered system and determine whether the corrective action satisfies predefined safety requirements. One important distinction is that detecting a failure does not automatically mean the system has successfully recovered. A recovery action might eliminate missing values while still leaving the model's predictive performance significantly degraded. This distinction is central to my research.
7. Why Credit-Default Prediction? For my initial experiment, I selected the UCI Default of Credit Card Clients dataset. The dataset contains 30,000 observations with information about customers' credit limits, demographic attributes, payment histories, billing amounts, and previous payments. The prediction task is binary classification: Class 0: No default in the following month. Class 1: Default in the following month.

I chose this dataset for several reasons.
First, credit-default prediction represents a meaningful application where model reliability matters. Incorrect predictions can have consequences for financial decisions.
Second, the dataset contains both numerical and categorical features, making it suitable for experimenting with multiple types of data-quality and distribution failures.
Third, its manageable size allows me to run controlled experiments, reproduce results, and compare recovery strategies without requiring extensive computing infrastructure.
Finally, the dataset provides a useful environment for separating three research questions:
Can the system detect a problem? Can it correct the problem? And does the correction preserve predictive performance?
Although the dataset provides a realistic classification problem, my experiments use historical data and simulated failures. They do not constitute a production credit-decision system.

1. Defining the Research Hypothesis A research project needs a testable hypothesis rather than a general expectation that automation will improve reliability. My primary hypothesis is: Controlled data-quality and distribution failures will produce measurable changes in classifier performance, and a diagnosis-guided recovery framework can mitigate some of those changes while enforcing explicit acceptance criteria. I also want to investigate whether detecting a data problem and correcting that problem are sufficient to restore model performance. For example, consider a batch of incoming records where 15% of the values in a previously complete feature become missing. The system might detect the increase in missingness and replace the missing values using a training-derived median. Technically, the missing values have been corrected. But has the model actually recovered? Not necessarily. The imputed values may not preserve the relationships that existed in the original data, potentially reducing predictive accuracy, recall, or F1-score. Therefore, recovery success must be measured through predictive performance, not merely successful data preprocessing.
2. Designing the Self-Healing Architecture The next step was designing the system's end-to-end workflow. My proposed architecture follows this sequence: Dataset → Data Profiler → Baseline ML Model → Failure Injection → Failure Detection → Diagnosis → Recovery Engine → Performance Evaluation → Accept/Reject Decision → Experiment Logging Each component has a distinct responsibility. The data profiler establishes a healthy reference. The baseline model provides a performance benchmark. The failure injector creates controlled abnormal conditions. The detection and diagnosis components identify the failure, while the recovery engine applies an appropriate correction. Finally, the evaluation layer determines whether the recovered system meets the required performance tolerance. This last stage is particularly important. A self-healing system should not automatically accept every attempted correction simply because the recovery code executed successfully. It needs a mechanism for rejecting unsuccessful recovery attempts. In a future production-oriented implementation, rejection could prevent a candidate from being promoted and trigger additional investigation. For the current experimental implementation, the accept/reject decision is an evaluation outcome rather than an actual production deployment.
3. Establishing a Healthy Baseline Before evaluating failures, I needed to specify how a healthy model would be trained and assessed. For Phase 1, I selected Logistic Regression as the initial classification algorithm. Why start with Logistic Regression instead of a complex neural network? Because the purpose of the first experiment is not to maximize predictive performance. It is to understand how controlled data failures influence model behavior and whether recovery strategies can mitigate that influence. Logistic Regression offers several advantages: It is relatively interpretable. It is computationally efficient. It provides a reproducible baseline. It makes comparisons across failure scenarios easier to understand.

My baseline pipeline uses StandardScaler for numerical feature scaling and Logistic Regression for classification.
I also planned a small TensorFlow/Keras multilayer perceptron as a future comparison model.
This will eventually allow me to investigate whether different model architectures respond differently to identical data failures.

1. Designing the First Failure Experiment One of the most important Phase 1 decisions was limiting the initial experiment to a clearly defined failure. I selected missing-value injection in the BILL_AMT1 feature. BILL_AMT1 represents the customer's bill statement amount for a particular month. The experiment introduces missing values at four controlled levels to evaluate how increasing data corruption affects model performance and recovery effectiveness. Missingness Injection Scenarios Healthy Reference (0%): No missing values are introduced, providing a baseline for comparison. Low-Severity Failure (5%): Missing values are introduced into 5% of the selected feature's observations. Moderate-Severity Failure (15%): Missingness increases to 15% to evaluate the model's response to more substantial data degradation. High-Severity Failure (30%): Missingness reaches 30%, representing a more severe data-quality failure.

These controlled scenarios allow me to systematically investigate how different levels of missingness in the BILL_AMT1 feature influence model performance, failure detection, and the effectiveness of the recovery strategy.
These scenarios provide a controlled way to examine how increasing missingness affects classifier performance.
Rather than changing multiple features simultaneously, I initially focus on one feature so that the failure mechanism remains interpretable.
This design makes it easier to compare outcomes and isolate the effect of the injected corruption.
The experiment is conducted on held-out validation data, while training-derived statistics remain fixed.
This separation is important for preventing information leakage.

1. Failure Detection and Recovery Strategy For the first experiment, I defined a simple detection rule. Trigger a missingness alert when the increase in missingness exceeds five percentage points relative to the healthy reference. This means that a five-percentage-point increase does not trigger an alert under the strict greater-than rule, while increases of 15 or 30 percentage points do. I chose this threshold as an initial experimental parameter, not as a universally appropriate production threshold. Future experiments may evaluate how different thresholds affect detection sensitivity and false alarms. For recovery, I selected median imputation using statistics calculated from the training data. Why median imputation? Financial variables such as billing amounts may contain skewed distributions and extreme values. Median imputation provides a simple, reproducible correction that is less sensitive to extreme values than mean imputation. However, median imputation is not automatically a successful recovery strategy. It replaces missing entries, but it cannot reconstruct the original information. That limitation makes it a useful starting point for evaluating whether a seemingly reasonable corrective action actually restores acceptable model behavior.
2. How Do We Measure Successful Recovery? This was one of the most important methodological decisions in Phase 1. I selected the following evaluation metrics: MetricPurposeF1-scorePrimary measure balancing precision and recallRecallMeasures how many actual defaults are identifiedPrecisionMeasures correctness among predicted defaultsROC-AUCMeasures ranking and discrimination abilityAccuracyProvides an overall classification measure Because the dataset has an imbalanced target distribution, accuracy alone may be misleading. A model can achieve relatively high accuracy by predicting the majority class while failing to identify many customers who actually default. For that reason, I selected F1-score as my primary evaluation metric, while retaining recall and ROC-AUC as important supporting measures. Defining a recovery safety rule I established an initial tolerance of a maximum 10% relative decline in F1-score compared with the healthy baseline. The relative performance change is calculated as: Relative F1 Change (%) = [(F1 Recovered − F1 Baseline) / F1 Baseline] × 100 For example, suppose the healthy baseline F1-score is 0.40. A recovered F1-score of 0.38 represents a 5% relative decline and would satisfy the tolerance. A recovered F1-score of 0.32 represents a 20% relative decline and would be rejected. These are illustrative values, not experimental results. This rule establishes a measurable boundary between an acceptable and an unacceptable recovery outcome. The 10% tolerance is an initial research design choice. It will need further validation, including sensitivity analysis and consideration of subgroup-level performance.
3. Why Reproducibility Matters A self-healing system must be evaluated under controlled conditions. Otherwise, differences in model performance might result from random sampling, inconsistent preprocessing, or changing training data rather than the injected failure itself. For my initial experiment, I specified: Stratified train/validation/test splitting. A fixed random seed of 42. Consistent preprocessing. Training-only reference statistics. Defined failure severity levels. Consistent evaluation metrics. Experiment logging for comparing outcomes.

The planned data split is:
70% training, 15% validation, and 15% testing.
Validation data supports controlled failure experiments and recovery assessment, while the held-out test set provides a separate resource for final evaluation.
I also intend to document unsuccessful recovery attempts rather than reporting only favorable results.
A rejected recovery is not necessarily a failed research experiment.
It can provide important evidence about the limitations of the correction strategy.

1. What Makes This Project Different From Ordinary ML Preprocessing? A reasonable question is: Isn't this just preprocessing with additional steps? Traditional preprocessing commonly applies predetermined transformations to data before prediction. A self-healing framework adds monitoring, explicit detection criteria, diagnosis-guided correction, and a decision mechanism that evaluates whether the correction is acceptable. For example, a conventional pipeline might apply median imputation whenever missing values appear. My proposed framework asks additional questions: Was the increase in missingness unusual? Which feature was affected? What correction should be attempted? Did the correction preserve predictive performance? Should the recovery outcome be accepted or rejected? These questions transform preprocessing from a fixed transformation into part of a monitored and evaluated recovery workflow. However, this does not mean that every automated preprocessing pipeline qualifies as a self-healing system. The meaningful distinction must be demonstrated experimentally through reliable detection, appropriate diagnosis, measurable recovery, and controlled acceptance decisions.
2. Challenges and Limitations Several challenges became clear while designing Phase 1. Defining meaningful degradation: A performance decrease is not automatically operationally significant. I need explicit metrics, thresholds, and justification for those thresholds. Separating detection from recovery: A system can correctly identify a failure without being capable of repairing it. Avoiding data leakage: Recovery statistics must be derived from training data rather than recalculated using held-out evaluation data. Handling different failure types: Missingness, numerical drift, and categorical shifts may require different detection and recovery strategies. Evaluating reliability beyond aggregate metrics: Future experiments should consider whether recovery affects different subgroups differently. Avoiding exaggerated automation claims: An automated imputation and evaluation workflow is not the same as a fully autonomous production system. These limitations will help guide subsequent phases.
3. What's Next? Phase 1 established the experimental foundation for my research. The next step is Phase 2: Healthy Data Profiling and Reference Development. In Phase 2, I will examine the dataset's actual characteristics, including missing values, duplicates, numerical distributions, unusual categorical codes, and class imbalance. The goal is to construct a healthy data reference that can support subsequent monitoring and failure detection. After that, I plan to implement the baseline classifier, inject controlled failures, evaluate recovery strategies, and compare accepted and rejected recovery outcomes. Longer-term extensions may include additional failure types, alternative model architectures, LLM-assisted diagnosis, retrieval-augmented generation (RAG), and cloud-based deployment using platforms such as Google Cloud or Microsoft Azure. These are planned extensions rather than completed capabilities. Final Thoughts The most important lesson from Phase 1 is that building a reliable machine learning system requires more than selecting an accurate model. It requires understanding how the system should respond when its underlying assumptions no longer hold. A model may perform well under healthy conditions but remain vulnerable to missing data, distribution shifts, and unexpected changes in its inputs. The real challenge is not simply detecting these failures but determining whether corrective actions genuinely restore acceptable model performance. One principle guides my research: successful recovery must be demonstrated through measurable evidence, not assumed simply because a corrective action was completed. Phase 1 provided the foundation for investigating this idea by establishing a structured experimental framework, defining failure scenarios, selecting evaluation metrics, and introducing recovery acceptance criteria. Through this series, I will document the technical decisions, experiments, challenges, and lessons learned as I develop my Self-Healing Machine Learning system. Coming Next: Phase 2 - Healthy Data Profiling and Reference Development In the next article, I'll explore the credit-default dataset, examine its underlying characteristics, and establish a healthy data reference to support automated failure detection. Can machine learning systems move beyond detecting failures toward making reliable, evidence-based recovery decisions? That's the question driving this research.