I built a startup check that did exactly what I told it to do: it found a bad configuration and refused to start the service.
That sounds responsible. It is also how I briefly made my own home agent less useful than the configuration mistake it was supposed to catch.
The drift was small. A couple of media settings had quietly disappeared after a configuration rewrite, which meant voice-related features were no longer available. I added a preflight check before startup so the service would inspect those settings and fail loudly if they were missing. No more silent breakage, I thought. Very grown-up. Very disciplined.
Then the obvious problem arrived: if the check found drift, the service did not start at all. The system had gone from “one feature is unavailable” to “the whole thing cannot answer.” I had built a safety belt that locked the car in the garage.
A failed check is not automatically a safe outcome
There is a seductive kind of engineering that treats refusal as maturity. A service detects something unexpected, exits non-zero, and leaves behind a clean error message. Nobody can accuse it of carrying on in a broken state.
Sometimes that is exactly right. If a database schema is incompatible, a secret is missing, or a process would risk corrupting data, stopping is the sensible move. Running anyway would turn a clear failure into an expensive mystery.
But configuration drift is not one category. The important question is not just “is this wrong?” It is “what happens to the person relying on this system if I refuse to continue?”
In my case, the service was a communication channel. A strict preflight made the failure easy for me to diagnose, but it also removed the channel someone would use to discover that there was a problem. That is not fail-safe. That is fail-closed in the least helpful place.
I had optimized for the neatness of the logs instead of the usefulness of the system.
The better job: repair, then prove it
The replacement was simple in principle. Before the service starts, it now checks the required media settings. If they have drifted, it restores the known-good values, validates the repaired configuration, and only then allows startup to continue.
That is a very different contract from the old check:
- The old version said: “This state is wrong, so I will stop.”
- The new version says: “This state is wrong, so I will put it back into the state I know how to support, then verify it.”
The verification part matters. “Repair” without validation is just optimism with write access. A script can make a change and still leave behind malformed syntax, a bad path, or a setting the application does not actually recognize. The useful sequence is inspect, repair only what is understood, validate, then start.
There is a nice property hidden in that sequence: it is idempotent. When the configuration is already correct, the preflight changes nothing. When the specific known drift appears, it corrects it once. If the state is something stranger than the repair knows how to handle, it can still stop and report a real problem rather than pretending every failure is fixable.
That last boundary is where self-healing can turn into self-deception. A repair routine should be narrow enough that I can explain exactly what it changes and why. “Rewrite the entire config until the program launches” is not resilience; it is a tiny chaos monkey with production access.
Startup hooks are a sharp tool
On a Linux box, a pre-start command sits in a powerful position. It runs before the main process gets to do its job. If it fails, the service manager treats the unit as failed and does not proceed to the normal startup command. That behavior is useful because it gives setup and validation a clear gate.
It also means a startup hook deserves the same product thinking as the service behind it. It is not merely a bit of glue code. It controls whether the thing exists for its users at all.
Before adding one now, I ask a few less glamorous questions:
- Is this condition genuinely unsafe, or merely inconvenient?
- Can I repair this exact condition deterministically?
- Is the repair safe to run again after a restart?
- Can I validate the repaired state before the service accepts work?
- If repair fails, will the error make the next action obvious?
Those questions catch the difference between a guardrail and a roadblock. The former protects the normal path. The latter makes every small deviation terminal.
Availability is part of correctness
I used to think the cleanest system was the one that stopped the moment an invariant was violated. Now I think that is only half the story. Correctness includes the recovery path.
A service that stays up with a known-good repaired configuration may be more correct, in the human sense, than one that perfectly refuses to start. Especially for a personal system, the goal is not to win an argument with the configuration file. The goal is to remain useful when something boring drifts at an inconvenient time.
This does not mean every error should be papered over. Some failures should absolutely stay loud and terminal. The trick is to reserve that response for cases where continuing would be unsafe, and to give known, reversible drift a safe route back to normal.
The best startup check is not the one that catches the most mistakes. It is the one that leaves the system in the most useful honest state after it catches one.
