cns me / blog Cloudy with a Chance of Freefall
← Index | | 12 min read

Never Trust a Human in Production

AI Generated: A parent at a party watches someone hold their baby, caught between composure and alarm

My youngest was about three months old when a colleague at a Christmas party asked if she could hold her.

I said yes. Of course I said yes. What sort of person says no to that? She was a perfectly warm, perfectly competent adult with her own children, her own history of not dropping babies. There was no rational reason to refuse.

And yet.

I stood there, wine glass in hand, watching my daughter in someone else's arms, and every nerve in my body was screaming. Not quietly. Screaming. The social norms of the situation had caged me. You cannot snatch your baby back from a competent person who is delighted to hold her. That is the kind of thing that ends friendships and generates talk for decades. So I stood there. Smiled. Made conversation. And felt like I was watching a car crash in slow motion that only I could see.

Nothing happened. My daughter is now far too old and far too opinionated to anyone to swaddle her. She is perfectly fine. All my children are. The anxiety was real. The danger was not.

But I have never quite shaken the feeling — and I suspect I am not alone in this — that the anxiety of watching someone else hold something precious is one of the most instructive experiences a human being can have. Because it turns out that feeling maps, almost exactly, onto the experience of watching a junior engineer with root access.

Here is what makes the anxiety interesting before we dismiss it as mere sentimentality. A Lancet study published in 2025 — a large one, 73,845 emergency department visits — found that parental concern is a better predictor of ICU admission than any vital sign. Not temperature. Not respiratory rate. Not blood pressure. The parent's worry. The instinct is not irrational. The parental alarm system, honed by millions of years of evolution, registers information that a blood pressure cuff cannot. The person closest to the child knows something the chart does not.

So the anxiety is real. The question is what it is measuring.

Article content
AI Generated: A victorian surgeon surveys his ward with total authority

In 1889, William Stewart Halsted designed the American surgical residency system. He was the first chief of surgery at Johns Hopkins Hospital, a man of extraordinary technical gifts, and he built a training programme of total, hierarchical control. The senior surgeon was not merely the most skilled person in the room; he was the only person in the room who mattered. Residents worked thirty-six-hour shifts, learning through exhaustion and submission. The system Halsted created would shape American medicine for a century.

What is less often discussed is that Halsted was, at the time, a functioning cocaine addict whose colleagues had repeatedly had to remove him from the operating theatre mid-procedure. The man who designed the system of total surgical control could not trust himself to remain present for an entire operation.

This is not a footnote. This is the story.

The same pattern appears, with striking consistency, across the history of human authority over complex systems. In 1986, Anatoly Dyatlov was the deputy chief engineer on duty at the Chernobyl nuclear power plant during the safety test that triggered the explosion. He had been awake for most of the preceding day, was operating under enormous political pressure to complete the test before the reactor was taken offline for maintenance, and overrode the safety protocols of engineers who raised objections. He knew better. He was the most experienced person in the room. He was certain.

228 people were on board Air France Flight 447 when it entered a high-altitude weather system over the Atlantic on 1 June 2009. The autopilot disengaged. The pilots, who had become so accustomed to the aircraft flying itself that they had, in the quiet clinical phrase of accident investigators, experienced "manual flying skill deterioration," could not hand-fly the aircraft back to stable flight. They had been in the cockpit for three and a half hours. The aircraft entered a stall. Everyone died.

And on 3 July 1988, Captain William C. Rogers III of the USS Vincennes shot down Iran Air Flight 655, killing all 290 civilians on board, including 66 children. The Aegis combat system had flagged the aircraft as a potential military threat. Rogers had centralised all decision-making authority on the bridge. No one in his command structure challenged the order. He received the Legion of Merit.

These are not stories about incompetence. Dyatlov was experienced. Rogers was decorated. The Air France pilots were trained and certified. Halsted was genuinely brilliant. They are stories about something more interesting: the confident, senior, authority-holding human being as the single largest source of risk in a complex system.

Then Like Now

The most dangerous driver on the road is, statistically, the one who believes most strongly that they are an above-average driver. Seventy-four percent of drivers rate themselves above average, a percentage that is arithmetically impossible but psychologically inevitable. Kruger and Dunning documented in 1999 what anyone who has managed people already knows: the twelfth-percentile performer self-rates at the sixty-second percentile. The most confident person in the room is statistically the most dangerous.

New parents lose approximately seven hundred hours of sleep in the first year. Seventeen hours without sleep produces cognitive impairment equivalent to a blood alcohol concentration of 0.05 per cent. The particularly cruel feature of severe sleep deprivation is that it impairs your ability to assess your own impairment. You feel fine. You feel capable. You feel like you are the most qualified person in the room to be holding this baby. And you are running on the neurological equivalent of three pints of lager.

The exhausted, overconfident person who refuses to let go is not the guardian of the system. They are the greatest threat to it.

I know this because I have been that person. Standing at the Christmas party, not entirely sober, running on the accumulated sleep debt of three months with a newborn, absolutely convinced that nobody else in the room was as well-qualified as I was to ensure my daughter's safety. The social norms that kept me from snatching her back were not cruelty. They were, in retrospect, a form of engineering. They were the guardrail.

Article content
AI Generated: A signalman at his levers watches a train approach

Engineering has known about this problem for a very long time.

In the 1880s, Frank Sprague was designing the electrical systems for street railways in Richmond, Virginia, when he encountered a problem that transit operators had been refusing to acknowledge: the human driver, fatigued, distracted, or dead, was a vehicle that would keep moving. In 1888, the dead man's switch entered service — a mechanism that required the driver to maintain active, conscious contact with the controls for the vehicle to continue operating. Release the control, and the vehicle stops. The philosophy was not that drivers were untrustworthy. The philosophy was that trust was irrelevant. The system should not depend on it.

The principle was reinforced with considerable violence after the Malbone Street Wreck of 1918, when a Brooklyn Rapid Transit motorman, operating well beyond his experience level after a union dispute had pulled the regular drivers, ran a train into a tunnel at full speed, killing 93 people. The dead man's switch became universal in mass transit. Not because every driver was dangerous. Because any driver could be.

Forty-four years later, in the wake of a Cuban missile crisis that had brought the world to the edge of thermonuclear war, President Kennedy signed National Security Action Memorandum 160, requiring all American nuclear weapons to be fitted with Permissive Action Links — cryptographic locks that required authorised codes before the weapon could arm. The Permissive Action Link did not reflect a belief that American military personnel were disloyal. It reflected something more honest: that no individual, regardless of rank, training, or loyalty, should be trusted with unilateral authority over a nuclear weapon. The two-man rule that accompanied it — no single person could arm a weapon alone — was the dead man's switch applied to the end of the world.

The inscription above the door of nuclear weapons engineering, if you were to read it plainly, would say: we do not trust anyone. Not even ourselves.

Article content
AI Generated: a PAL panel with two keyholes too far apart for one person

The technology industry arrived at the same conclusion, more slowly and with considerably more human error along the way.

In July 2024, a single developer's code update to CrowdStrike's Falcon sensor crashed 8.5 million Windows systems simultaneously. Hospitals cancelled surgeries. Airports grounded flights. Emergency services lost communications. The developer did not intend any of this. The developer was, almost certainly, competent. The developer was, almost certainly, confident. The developer had the equivalent of a master key to nearly every protected Windows machine on earth, and there was no dead man's switch, no two-man rule, no Permissive Action Link between their keystroke and global infrastructure. The Uptime Institute's 2025 analysis found that 40 per cent of organisations had experienced a major outage attributable to human error in the preceding three years.

The GitOps philosophy that has been gradually displacing the older model of direct server access encodes something important: no human types a command directly into a live system. All changes flow through version control. All changes are reviewed. All changes are auditable. The system does not depend on any individual's judgement, confidence, or wakefulness at three in the morning when the pager goes off. The philosophy is not that engineers are untrustworthy. It is that trust is beside the point.

Call it the constraint ladder. There are, broadly, four levels at which we trust a human with something precious. You hold it yourself. You hover while they hold it. You leave the room. You leave the house. Each level removes a layer of direct control and adds a layer of structural safety. The dead man's switch is leaving the house — the driver is designed out entirely. The Permissive Action Link is leaving the room — a human is present but cannot act alone. A GitOps review pipeline is hovering — the engineer is in the loop but constrained by process. Most of us, most of the time, are stuck at level one: holding the baby ourselves, too tired to do it safely, too anxious to hand it over. The question is not whether to climb the ladder. It is which rung is right for what you are holding.

Human confidence in human authority is not, however, without its own ironies.

Lisanne Bainbridge published a paper in 1983 that has now accumulated more than 1,800 academic citations. She called it "Ironies of Automation," and its central observation is one of the more uncomfortable findings in the literature of human factors engineering: the guardrails that protect us from the consequences of human error also atrophy the human skills that would be needed if the guardrails failed. Automate the routine operation, and the operator never practises routine operation. Which means that when the automation fails — and it will fail — the human who is supposed to take over has been systematically deprived of the practice required to take over safely. Seventy-seven per cent of commercial pilots report that their manual flying skills have deteriorated since autopilot became standard. Air France 447 was not only a story about pilots who could not hand-fly. It was a story about a system that had trained them not to.

But the constraint ladder has a cost at every rung, and this is the dimension that Bainbridge's irony points toward that we rarely follow to its conclusion. Each level that removes the human from the decision also removes the human's ability to take over when that level breaks. So the question to ask at each rung is this: is human recovery possible when this constraint fails? If yes — if the aircraft needs a pilot when the autopilot disengages, if the pipeline needs a human when the automated rollback loops — then skill maintenance is an active design requirement, not an afterthought. It must be budgeted, scheduled, practised. The two-man rule for nuclear weapons is the honest answer to the other case: when no human will ever need to act unilaterally, full structural constraint is not a compromise but a clarity. Bainbridge's irony applies to recoverable failures. It does not apply to all failures equally. The important engineering question is not whether to constrain human authority. It is whether, at the rung you are designing for, a human ever needs to come back.

This is where the Christmas party anxiety becomes genuinely interesting. Not because the anxiety was wrong — the Lancet finding is evidence that it was tracking something real — but because of what it was tracking. The anxiety was not information about her safety. It was information about mine. About my need to be the one holding her. About the discomfort of distributed trust.

If you have ever hovered over a colleague's deployment, or insisted on reviewing every pull request yourself, or been the person who stays on the call long after everyone else has confirmed the fix — then you have been at that Christmas party. You know the feeling. The question is whether you are holding on because the system needs you, or because you need to be the one holding.

The organisations that have best navigated this tension — between the genuine danger of unconstrained human authority and the genuine value of human judgement — are the ones that have built their constraints into the system rather than relying on the virtue of individuals. Kennedy's nuclear protocols did not depend on military personnel being good people. The GitOps pipeline does not depend on engineers having good days. The dead man's switch did not depend on drivers being attentive. The constraint is structural. The trust is an emergent property of the structure, not a prerequisite for it.

Designing humans out of the loop entirely is not the answer either. Designing the right constraint at the right rung — consciously, with full awareness of what the constraint costs as well as what it protects — is the answer. The parent who never lets anyone hold the baby is not keeping the baby safe. They are keeping themselves comfortable. And the organisation that removes every human from every production decision has not solved the trust problem. It has moved the trust to the automation, where it is invisible.

William Halsted, who built a residency system on the premise that total hierarchical control was the only safe model for surgical training, spent years of that same residency disappearing from operating theatres he could not finish. The system he designed to eliminate human error was administered by a human whose errors were its founding contradiction. The engineers who will spend 2026 arguing loudest for unconstrained production access are the ones least likely to recognise their own Dyatlov moment when it arrives — rested, confident, certain, and about to override the safety protocol.

Never trust a human in production.

Including yourself.

(Views in this article are my own.)

🦩