Self-Healing
Self-Healing in Three Layers
The system is designed to start recovering on its own, not only to detect problems. It reacts to two kinds of trouble: running short of resources, and errors inside one of its parts. This page describes the design. The layers are built and certified, which is not a claim that every part runs continuously at all times.
The Complete Self-Healing Loop
The Design
When the system detects a problem, it responds in one of two ways.
Running short of resources (disk, processor, memory)
- A limit is crossed and survival mode starts
- Heavy services are shut down in an orderly way
- Critical services keep running
- When the pressure eases, the services start again automatically
- The aim: the system stays up, recovers and carries on
An error inside one of its parts
- The error is written to a log automatically
- Errors are collected in one place, so they are visible
- A monitoring agent notices the error on its next check
- A repair request is sent to the agent responsible for that part
- That agent works on the fix and reports back; some problems are sent to a person instead
- The aim: the system corrects itself, and the same error becomes less likely to return
The Three Layers
Layer 1: Survival. Detect a resource crisis, protect the critical services, and wait for recovery. The check runs about once a minute.
Layer 2: Visibility. Every error is visible in one shared place. No error is hidden in a log file.
Layer 3: Repair. The monitoring agent finds problems and sends repair requests automatically.
What This Means
Without self-healing:
- The system fails
- Someone has to notice
- Someone has to work out the cause
- Someone has to fix it
- Recovery time is unpredictable
With this design, when resources run short:
- Survival mode starts at once
- Heavy services stop, and critical ones continue
- Services restart when the pressure eases
With this design, when a part reports an error:
- The error is captured and made visible automatically
- The monitoring agent notices it
- A repair request is sent to the responsible agent
Result: recovery time depends on automation, not on someone noticing.
Rules for New Components
Every new capability must write standard logs. This is not optional; it is the contract.
New component checklist:
- ✅ Writes its log in the standard place
- ✅ Uses one line of structured data per entry
- ✅ Includes the time, the component, the severity, the type of error, the message and the stack trace
- ✅ The log collector can read it
- ✅ The monitoring agent can find its errors
Without this contract: a component crashes and nobody notices. With this contract: a component crashes and the system can start a repair.
Consequence
This is not just a feature. It is a redesign of how reliability works.
In traditional systems, the time to repair depends on human intervention. Here it depends on automation: better automation means faster recovery.
The design lets recovery improve over time as the system learns from what has gone wrong before. That comes from recorded experience, not from retraining.
How It Works
These pages explain how each claim is designed to work. We are adding evidence you can look at yourself, one proof at a time.