Kubernetes 自我修復基礎設施指南：探針、自動擴縮、GitOps 與自動修復解析

Last updated: October 30, 2026

Modern applications fail in small ways all the time: a container crashes, a node becomes unavailable, a deployment ships a bad configuration, or traffic spikes faster than a fixed number of replicas can handle.

Self-healing infrastructure is how you stop each of those small failures from becoming an outage a human has to wake up for.

Kubernetes gives you a strong foundation — it restarts containers, replaces pods, and reschedules work when nodes fail. But that foundation alone is not a complete self-healing system. Production teams layer autoscaling, GitOps reconciliation, observability-driven alerts, and carefully limited auto-remediation on top of it, and then they test that the healing actually works.

This guide walks through each layer in order, with practical examples you can adapt to a test cluster:

1. What “self-healing” actually means (and what it does not)
2. The healing Kubernetes already does for you
3. Probes: the signals healing depends on
4. Autoscaling as healing: HPA and KEDA
5. GitOps: drift correction and safer rollbacks
6. Operators and node-level repair
7. Service mesh: retries, timeouts, and circuit breaking
8. Observability and alert-driven remediation — with guardrails
9. Chaos engineering: proving recovery works
10. Security-related healing, done carefully
11. AI-assisted operations: where it helps, where it must not act alone
12. A step-by-step blueprint and the mistakes to avoid

My interest in this started from the opposite experience — my own 3D portfolio deployment, where errors piled up during uploads and every fix was manual. The gap between that and a system that detects, replaces, and verifies on its own is what this guide is about.

Originally published on DevOps Enginer.

1. What is self-healing infrastructure?

Self-healing infrastructure is a system that can:

• Detect a failure or unhealthy state (from probes, metrics, logs, events, or traces)
• Decide whether a known, safe corrective action applies
• Act — restart, replace, reschedule, scale, roll back, or restore the desired state
• Verify that the system is healthy again, and alert a human if it is not

That last step matters. Automation that acts but never verifies — and never escalates — is not healing, it is just hidden risk.

What self-healing is not

• It is not “no on-call ever.” Novel failures still need humans.
• It is not a reason to skip root-cause analysis. A pod that restarts every hour is being healed and is also telling you something is wrong.
• It is not safe to automate every fix. Start with reversible, low-blast-radius actions (replace one pod, scale within limits), not irreversible ones (delete data, change shared infrastructure).

The layers at a glance

2. The healing Kubernetes already does

Before adding any tool, make sure you are actually using what Kubernetes provides by default.

ReplicaSets and Deployments: maintaining the desired number of pods

A Deployment declares how many replicas should exist. If a pod is deleted or a node fails, the ReplicaSet controller works to create replacements so the actual state matches the desired state.

Check it yourself in a test cluster:

# Verify these commands on your cluster/version before publishing
kubectl get deployment my-app
kubectl get pods -w
kubectl delete pod <one-pod-name-from-the-deployment>
# Watch a replacement pod being created automatically

Rescheduling after node failure

If a node becomes NotReady, Kubernetes will eventually evict pods (subject to tolerations and PodDisruptionBudgets) and schedule replacements on healthy nodes. Stateful workloads need extra care — persistent volumes, ordering, and identity are why StatefulSets exist. Do not assume a stateful service heals the same way a stateless one does; test it.

PodDisruptionBudgets: healing without taking everything down at once

A PodDisruptionBudget (PDB) limits how many pods of an application can be voluntarily disrupted at the same time — for example during a node drain or a rolling update.

# Example only — verify apiVersion and fields on your cluster
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-app-pdb
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: my-app

A PDB does not prevent involuntary disruptions (a node physically failing), and a badly set PDB can block legitimate maintenance such as node upgrades. Treat it as a safety rail, not a guarantee.

3. Probes: the signals healing depends on

Probes are the most important — and most commonly misconfigured — part of Kubernetes self-healing.

The classic mistake is using a liveness probe for a dependency check. If your liveness probe fails whenever a database is briefly unavailable, Kubernetes will restart every pod — turning a small dependency problem into a restart storm. Dependency checks usually belong in readiness, not liveness.

# Example only — verify paths, ports and timings for your app
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
        - name: my-app
          image: example/my-app:1.0.0 # replace with your own image
          ports:
            - containerPort: 8080
          startupProbe:
            httpGet:
              path: /healthz
              port: 8080
            failureThreshold: 30
            periodSeconds: 2
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
            periodSeconds: 10
            failureThreshold: 3
          resources:
            requests:
              cpu: "250m"
              memory: "256Mi"
            limits:
              cpu: "500m"
              memory: "512Mi"

Practical rules:

• Make probe endpoints cheap — they should not run a heavy query on every check.
• Give slow apps a startup probe instead of inflating liveness delays.
• Set resource requests honestly. Autoscaling and scheduling decisions are only as good as these numbers.

4. Autoscaling as healing: HPA and KEDA

Sometimes “healing” is not about a broken pod — it is about a healthy service that no longer has enough capacity.

Horizontal Pod Autoscaler (HPA)

HPA adjusts replica count based on metrics such as CPU or memory utilisation relative to the requests you set.

# Example only — verify on your cluster; requires a metrics provider
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  minReplicas: 3
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

If your pods have no resource requests, percentage-based HPA has nothing meaningful to calculate against. That is one of the most common reasons “HPA does nothing.”

KEDA: scaling on external signals

KEDA (Kubernetes Event-Driven Autoscaling) extends scaling to signals HPA does not natively use — for example a Kafka consumer lag, a queue depth, or a schedule. It can also scale a workload to zero when there is no work, which plain HPA cannot do for the target itself in the same way.

# Illustrative example only — trigger fields differ by scaler/version.
# Verify against the KEDA docs for the version you install.
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: worker-scaler
spec:
  scaleTargetRef:
    name: my-worker
  minReplicaCount: 1
  maxReplicaCount: 10
  triggers:
    - type: kafka
      metadata:
        bootstrapServers: kafka.example.internal:9092
        consumerGroup: my-group
        topic: jobs
        lagThreshold: "50"

HPA or KEDA? Use HPA for CPU/memory-driven web services to start. Reach for KEDA when the real signal is outside the pod — messages waiting, jobs queued, events arriving. Many teams use both: KEDA manages the ScaledObject, and it creates/manages an HPA behind the scenes.

Autoscaling is not predictive by default. It reacts to a signal crossing a threshold. Some cloud services and third-party tools offer forecasting features, but evaluate them on your own traffic patterns rather than assuming a product will predict your spikes accurately.

5. GitOps: drift correction and safer rollbacks

GitOps makes Git the declared source of truth for what should be running. A GitOps controller (such as Argo CD or Flux) continuously compares the live cluster with Git and can re-apply the desired state when they differ.

That reconciliation is a form of healing:

• Someone manually edits a Deployment in the cluster → the controller restores the version from Git.
• A configuration value drifts → the next sync brings it back.
• A bad change is merged → reverting the commit in Git is the rollback mechanism.

# Argo CD example only — verify fields for your Argo CD version.
# Warning: selfHeal will revert manual changes made outside Git.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/example/my-app-config.git # replace with your repo
    targetRevision: HEAD
    path: k8s/overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: my-app
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

A safer rollback pattern

Automatic rollback sounds attractive, and tools and rollout controllers can help automate parts of it — but a blind “if degraded, roll back” rule can also roll back during a dependency outage that has nothing to do with your release.

A safer pattern most teams can actually operate:

1. Detect: health checks / error-rate alerts fire for the new version.
2. Stop the spread: pause the rollout or halt promotion (do not push it to more clusters/regions).
3. Decide with data: is the failure correlated with the release, or with a shared dependency?
4. Roll back via Git: revert the change commit, let the GitOps controller sync the previous known-good state.
5. Verify and record: confirm health returns, then write up what failed so the same release is not automatically retried.

For a deeper comparison of this model with classic push pipelines, read our guide to GitOps vs Traditional CI/CD — it covers where GitOps helps recovery, and where it does not replace a good pipeline.

My own experience is with Argo CD rather than Flux, so the examples in this section lean towards Argo CD.

6. Operators and node-level repair

Operators: codifying operational knowledge

An operator is a controller built for a specific application or platform task — handling backups, upgrades, failover, or repair steps that a generic Deployment controller does not understand. Good operators encode a runbook a human would otherwise follow manually.

Well-known patterns in this space include node problem detection (surfacing disk, kernel, or hardware issues as node conditions/events) and automated, controlled node reboots for kernel updates — for example with tools such as Kured, which cordons and drains a node before rebooting it when a reboot is required. Verify the current maintenance status and documentation of any tool before recommending or installing it.

Guardrail: an operator with broad permissions that takes destructive action is itself a production risk. Scope its RBAC tightly, test upgrades in staging, and make sure you can pause/disable its automated actions quickly.

Node repair on managed Kubernetes

Managed services (EKS, AKS, GKE) typically offer some combination of managed control planes, node auto-upgrade, and node auto-repair/replacement for nodes that stay unhealthy. The exact features, defaults, and names differ by provider and change over time, so:

• Check the current documentation for your provider and your node pool settings.
• Confirm whether auto-repair is actually enabled — it is not safe to assume it is on.
• Make sure PDBs, topology spread, and multiple replicas are in place, or node replacement can still cause an outage for a single-replica workload.

On self-managed clusters, cluster autoscaling tooling (such as Karpenter or Cluster Autoscaler, depending on your platform) handles provisioning and replacing nodes based on pending pods and node health signals — again, verify the current tool that fits your cloud and version.

7. Service mesh: retries, timeouts, and circuit breaking

A service mesh can heal at the request level — the layer users actually feel:

• Retries for transient failures (with limits and backoff, not infinite retries)
• Timeouts so a slow dependency does not tie up every caller indefinitely
• Circuit breaking / outlier detection so hosts that keep failing are temporarily removed from load balancing
• Progressive traffic shifting so a new version receives a small share of traffic first

# Istio illustration only — verify for your mesh and version.
# Retries without timeouts and budgets can amplify an outage.
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: my-app
spec:
  hosts:
    - my-app
  http:
    - route:
        - destination:
            host: my-app
      timeout: 5s
      retries:
        attempts: 3
        perTryTimeout: 2s
        retryOn: 5xx,reset,connect-failure

The warning that matters: retries multiply load. If a downstream service is already struggling, aggressive retries from every caller can finish it off. Always pair retries with timeouts, a small attempt count, and ideally a retry budget. If you do not run a mesh, you can (and should) implement the same ideas in client libraries or an API gateway — the pattern matters more than the product.

Architecture choices also affect how much of this you need. Stateful, long-running container platforms and event-driven serverless services fail and recover differently — see our comparison of Serverless vs Containers for when each model simplifies (or complicates) recovery.

8. Observability and alert-driven remediation — with guardrails

You cannot heal what you cannot see. The minimum useful signal set is:

• Metrics for saturation and error rates (Prometheus or your cloud equivalent)
• Logs you can search centrally during an incident
• Kubernetes events for scheduling, probe, and eviction clues
• Traces when a request crosses several services (where you have them)

Example: alert on a restart loop, do not just silently restart forever

# Prometheus Operator example only — verify CRD, labels and
# expression against your installation before use.
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: my-app-alerts
  labels:
    release: prometheus # must match your Prometheus rule selector
spec:
  groups:
    - name: my-app.rules
      rules:
        - alert: PodRestartingFrequently
          expr: increase(kube_pod_container_status_restarts_total[15m]) > 5
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "Pod {{ $labels.pod }} is restarting frequently"
            description: "Investigate the root cause instead of relying on restarts to heal this workload."

Notice what this alert does: it pages a human to find the root cause. Not every alert should trigger an automatic fix.

If you automate remediation, limit it

Tools such as Alertmanager webhooks, event-driven automation, or runbook automation platforms can connect an alert to an action. Keep automated actions:

• Narrow: one known failure, one known fix (e.g. clear a known stuck state, restart one specific stateless workload)
• Idempotent: running it twice should not make things worse
• Rate-limited: a cooldown so it cannot loop every minute
• Observable: every automated action is logged and notified to a channel a human reads
• Reversible / stoppable: a kill switch exists and the team knows where it is

If an action does not meet those tests, automate the diagnosis instead: collect logs, events, and recent changes into the alert, and let a human decide faster.

9. Chaos engineering: proving recovery works

Self-healing you have never tested is a hope, not a capability.

Start small, in staging, with a clear hypothesis:

1. Hypothesis: “If one pod of my-app is killed, a replacement becomes Ready and error rate returns to normal without manual action.”
2. Blast radius: one pod, one non-production namespace, during working hours, with someone watching.
3. Experiment: kill the pod (tools such as Chaos Mesh or LitmusChaos can automate this, or do it manually with kubectl delete pod in a test environment first).
4. Measure: time to recovery, whether alerts fired correctly, whether the PDB/autoscaling behaved as expected.
5. Fix what you learn: most first experiments reveal a missing probe, a single replica, or an alert nobody saw — fix that before increasing the blast radius.

Only move experiments toward production gradually, with abort conditions defined in advance. Never run chaos experiments on systems or data you do not have explicit permission to test.

The closest thing to a chaos test in my own work was unplanned: repeated 3D portfolio uploads created real disorder, and I learned which parts were fragile by breaking them accidentally. A deliberate staging test is the grown-up way to get the same lesson.

10. Security-related healing, done carefully

Some recovery is security-driven:

• Replace a pod/node you suspect is compromised rather than trying to clean it in place
• Rotate credentials and certificates automatically before expiry causes an outage (and alert well before expiry)
• Detect runtime anomalies (for example with eBPF-based runtime security tooling such as Falco) and raise a high-quality alert
• Re-apply network policies and approved configurations via GitOps when they drift

Be conservative with automated blocking or quarantining in production. A false positive that automatically isolates a healthy payment or authentication service is an outage you caused yourself. Start in alert-only mode, tune, and automate enforcement only for cases with a very low false-positive rate — and always with a rapid override.

Never publish real secrets, internal addresses, or security configurations from a real environment in examples. All examples in this post use placeholder names for that reason — keep it that way in your own additions.

11. AI-assisted operations: where it helps, where it must not act alone

AI and anomaly-detection features are increasingly offered inside observability platforms — grouping related alerts, spotting unusual metric patterns, summarising logs, and suggesting likely causes. Used well, that shortens the detection and diagnosis steps, which is often where incidents lose the most time.

A sensible division of labour in 2026:

Treat an AI suggestion like a junior engineer’s suggestion: potentially very useful, always verified against the actual system state before anyone acts on it. Do not give an automated agent broad production credentials to “fix things” without scope limits, approval steps, and a full audit log.

12. A step-by-step blueprint you can start this week

You do not need every tool in this article. For most Kubernetes services, this order gives the most reliability for the least new complexity:

1. Fix the basics: 3+ replicas for important stateless services, honest resource requests/limits, correct readiness/liveness/startup probes, and a sensible PDB.
2. Add HPA for your main web-facing services. Add KEDA only where an external signal (queue, lag, schedule) is the real driver.
3. Put desired state in Git with Argo CD or Flux, and make rollback = revert a commit a practised, documented move.
4. Centralise the signals: one place for metrics, logs, and events, with a small set of alerts that each have an owner and a next action.
5. Automate one safe remediation — just one — with rate limits, notifications, and a kill switch. Expand only after it has behaved well.
6. Schedule a monthly game day / chaos test in staging, and track time-to-recovery like any other SLO-related metric. (Example: if your service has an illustrative SLO target, measure recovery against your own target — do not copy another company’s number.)
7. Review restarts and auto-actions weekly: anything healing repeatedly is a bug to fix, not a success to celebrate.

Common mistakes that break self-healing

• Liveness probes that check downstream dependencies → restart storms
• Single-replica “important” services → nothing for Kubernetes to heal to
• No resource requests → HPA and scheduling decisions become guesswork
• GitOps self-heal enabled before the team understands it → confusion when manual fixes keep disappearing
• Retries with no timeouts or budgets → cascading failures
• Automated remediation with no cooldown, logging, or kill switch
• Chaos tests skipped → discovering in a real incident that recovery never worked
• Treating restarts as “fixed” and never doing root-cause follow-up

Conclusion

Self-healing infrastructure is not one product, and it is not magic. It is a stack of unglamorous, well-tested practices:

Kubernetes controllers replacing what failed, probes giving honest health signals, autoscaling adding capacity when load is the problem, GitOps restoring the state you declared, a mesh or client protecting requests in flight, observability telling you the truth, and chaos testing proving the whole thing works before a real incident tests it for you.

Start with the layer closest to your most common failure, automate only what is safe and reversible, and add your own operational experience as you go. That is what turns a collection of tools into a system that genuinely recovers — and a team that can sleep a little better.

Next reads in this series:

• GitOps vs Traditional CI/CD — how Git-based delivery changes deployment, drift correction, and rollback.
• Serverless vs Containers — how your compute model changes scaling, failure modes, and recovery.

Author's note: I am currently improving the deployment setup behind my own 3D portfolio — the project that taught me most of the lessons in this guide. If you are experimenting with self-healing patterns yourself, I would like to hear what worked — reach me via the contact page.

Related guides

• GitOps vs Traditional CI/CD — the deployment model behind the drift correction described in section 5.
• Serverless vs Containers — how the compute model you choose changes failure handling and recovery.

FAQ

What is self-healing in Kubernetes?

Self-healing in Kubernetes means the platform automatically returns a workload to its desired state after a failure — for example restarting a failed container, replacing a pod, rescheduling pods after a node failure, or scaling replicas up when load rises. Higher layers such as GitOps and autoscaling extend that basic behaviour.

What is the difference between a liveness probe and a readiness probe?

A readiness probe decides whether a pod should receive traffic. If it fails, the pod is removed from Service endpoints but is not restarted. A liveness probe decides whether a container is still alive. If it fails, the container is restarted. Using a liveness probe for dependency checks is a common cause of restart loops.

Does Kubernetes heal itself automatically by default?

Partly. With a Deployment, Kubernetes will maintain the desired replica count and restart containers according to their restart policy and probes. It does not by itself fix a bad release, a misconfiguration in Git, an overloaded service with no autoscaling, or a failing downstream dependency — those need the extra layers described in this guide.

Should I use HPA or KEDA?

Start with HPA if CPU or memory utilisation is a good signal for your service. Use KEDA when the work is driven by an external signal such as a message queue, Kafka lag, or a schedule, or when you need to scale a worker to zero when there is no work. Many teams use KEDA alongside HPA rather than instead of it in every case.

Is GitOps the same as self-healing?

GitOps is one part of it. A GitOps controller healing configuration drift by restoring the state declared in Git is a form of self-healing. It does not replace probes, autoscaling, observability, or incident response for failures that are not configuration drift.

How do I test whether self-healing works?

Run a controlled experiment in a staging environment: state what should recover and how fast, kill one pod, then measure whether a replacement becomes ready, whether alerts fire, and whether users (or your test traffic) are affected. Fix what the test reveals and repeat regularly. Only expand toward production with small blast radii and clear abort conditions.

Can AI fix Kubernetes problems automatically?

AI-assisted tools can help detect anomalies, summarise logs, correlate alerts, and suggest causes or runbooks. Letting an AI system take high-impact production actions without deterministic limits, human approval for risky steps, and a full audit log is not recommended. Use AI to speed up diagnosis first; automate only narrow, reversible actions.

What should I automate first?

The safest first automation is usually already built in: correct probes, multiple replicas, and HPA within clear min/max limits. After that, pick one frequent, well-understood, reversible remediation, add rate limits and notifications, and observe it before automating anything else.