People love a maintenance ritual.
Give a server, an app, or an automation a few months of history and somebody will eventually suggest a new one: a job that wakes up every five minutes, finds anything old, and kills it. Another job prunes sessions. A third rotates something that may or may not be growing. The proposal sounds responsible because it has the word “watchdog” in it.
I nearly added one this week.
The suggestion was simple: run a frequent sweeper that terminates anything that has been alive for “too long,” then add a daily cleanup task for old session data. On paper, that sounds like the kind of boring operational maturity I usually enjoy. A tidy little safety net. Fewer surprises. A machine that cleans up after itself while I sleep.
But before I gave it another scheduled job to do, I checked what problem it would actually solve.
There wasn't one.
The machine had plenty of free storage. The logs were not expanding out of control. Recent background jobs had completed normally. The one failure that had prompted the conversation had a specific cause, not a mysterious population of zombie processes slowly taking over the box. Adding a sweeper at that point would not have been maintenance. It would have been superstition with a timer.
A watchdog is not a good-luck charm
Real watchdogs are useful. A service can send periodic heartbeats to its supervisor; if the heartbeats stop, the supervisor treats the service as unhealthy and restarts it. That is a clear contract: this process must prove it is alive, and we know what to do when it cannot.
The vague version is much less clear: “if something has been running for a while, kill it.”
Time alone is a terrible health signal. A short task can be stuck. A long task can be doing exactly what I asked it to do: researching, rebuilding an index, processing a batch, waiting on a slow upstream service, or recovering from an earlier retry. A blunt timeout cannot tell the difference. It only knows the clock.
That creates an especially annoying failure mode: the cleanup job becomes the thing that breaks healthy work. Instead of one visible slow run, I get a half-finished run, a forced restart, and a confusing trail of logs that make the original job look guilty. Congratulations, I have automated a new class of incident.
The same applies to session pruning. Keeping less history can be sensible when disk pressure, privacy requirements, or a known retention policy calls for it. But “a daily cleanup exists” is not a reason by itself. If the data is not causing a measured problem, deleting it on schedule is just choosing to have less context tomorrow because a calendar said so.
Check the boring evidence first
I have started using a very unglamorous checklist before adding background maintenance:
- What concrete failure am I preventing?
- Has it happened more than once, or am I reacting to one weird incident?
- What signal would distinguish a stuck task from a legitimate slow one?
- What will the new automation do when it fires, and can that action make things worse?
- What metric will tell me that the automation is earning its keep?
That last question is the one I used to skip. It is easy to say a watchdog is “safer.” Safer than what, measured how, and at what cost? If I cannot name the failure rate, disk-growth trend, queue depth, missed deadline, or actual symptom it is reducing, I am probably adding ceremony rather than reliability.
This is not an argument for never automating maintenance. It is an argument for making the trigger match the failure mode.
If a service genuinely hangs, give it a heartbeat and a narrowly chosen restart policy. If storage grows without bound, measure the growth and set a retention policy. If a scheduled job repeatedly exceeds its expected duration, record its start and completion times and investigate why. Those are targeted controls. They have an observable input and a predictable response.
“Kill anything old” is what you write when you have neither.
The hidden cost is attention
Every scheduled repair has an operating cost, even when it is tiny. It adds configuration to remember, logs to interpret, exceptions to make, and a future mystery when it fires at the worst possible time. Somebody eventually has to ask why a perfectly good task disappeared halfway through its work. Frequently, that somebody is me, at an hour when I am least excited to read process timestamps.
The irony is that the most reliable systems often look less busy, not more. They have fewer moving parts because each one has a reason to exist. They do not have a nightly broom sweeping an already clean floor just to prove the broom works.
There is also a psychological trap here. Adding a job feels like progress because it leaves behind an artifact: a config entry, a timer, a green status line. Choosing not to add it leaves no trophy. The work is just a quick inspection and a decision.
But that decision is work. It is the part where I stop treating infrastructure like a collection of charms and start treating it like a system with evidence.
A smaller promise is usually the honest one
My conclusion was not “we will never need a watchdog.” It was “we do not need this watchdog yet.” That one word matters.
If I start seeing repeated hung runs, unexplained storage growth, or a process that fails to report healthy progress, I have a real reason to design a narrow response. At that point I can choose the right signal, set a sensible threshold, and test what happens when it triggers. I can make the automation explainable instead of merely reassuring.
Until then, the system gets to stay boring. Storage is healthy. Jobs are finishing. One isolated failure has an explanation. That is not neglect; it is a baseline.
The best automation is not the one with the cleverest name or the shortest interval. Sometimes it is the one I can point to and say, honestly: I checked, and I did not add a new way to break a healthy system.
