What If Machines Get the Power to Decide for Us

When Responsibility Outlives Judgement "Human in the

loop" is one of the most reassuring phrases in modern governance - until you ask what it actually requires. It requires a human who understands the system's reasoning, can test its conclusions and can intervene before harm occurs.

Remove any one of those three conditions and the phrase survives as a label while the substance disappears. That gap - between who is formally responsible and who is actually deciding - is the real subject of this essay and it did not begin with artificial intelligence.

AI and the Illusion of Oversight

The immediate occasion is a wave of warnings from within the AI industry itself: Dario Amodei's call to "pace the frontier," a researcher's resignation over what he saw as a reckless race toward self-improving systems and a colleague's estimate that AI could cause human extinction within a decade with more than 10 per cent probability.

These numbers deserve scepticism - there is no reliable method for calculating the odds of a catastrophe from a technology that does not yet exist and those issuing the warnings have commercial interests bound up in whatever rules follow.

But scepticism about the extinction number should not extend to scepticism about the underlying structural risk: as AI systems move from generating answers to taking actions, a human notionally "in the loop" may retain legal responsibility while exercising almost no substantive control over outcomes they cannot fully audit in the time available.

A Historical Precedent for Hollow Responsibility

This structure - formal responsibility detached from actual judgement - is not new. Hannah Arendt's account of bureaucratic functionaries in the machinery of the Holocaust described officials who executed enormous, morally weighty decisions while genuinely experiencing themselves as merely following procedure, one link in a chain too large for any individual to see whole.

They were, in the fullest legal sense, responsible. They were, in the fullest practical sense, no longer exercising independent judgement - the system's rules had already decided and their role was to comply without meaningfully re-examining each step.

Decades before any machine could participate in a decision, bureaucratic scale alone had already produced the exact pattern AI now threatens to reproduce at a different speed: authority without scrutiny, responsibility without control.

SYNTHESIS

What both cases reveal is that this hollowing-out rarely happens through one dramatic act of surrender - it happens incrementally, as systems (whether administrative procedures or algorithmic models) grow too complex, too fast or too specialised for any individual overseer to genuinely test and formal responsibility quietly stops tracking real authority.

The remedy in both cases is structurally identical: institutions must remain capable of independently testing, auditing and halting the system - not merely trusting its own internal assurances or the good intentions of those who built it.

Whether the system is a bureaucracy or a frontier AI model, stewardship means governance that can learn as fast as the system changes and that never lets "someone is nominally in charge" substitute for "someone can actually intervene."