An argument built on the case studies, and the sequel to the Costume. That post found the levers run with the mysticism stripped off. This one removes something else: the person. The oracle can be a system, and it still cannot be argued with.
Status. The three systems, their failures, the inquiries and the remedies are documented and — in every case — formally adjudicated; the reading of them through this framework is an inference, offered as one. It names institutions, not individuals: where people are under investigation, the post says so and stops. As everywhere: a description of a system is not a verdict on anyone, and this post is not an argument against measurement.
The witch-and-the-debt post located the first lever precisely: not a belief in hidden agency, but an unfalsifiable slot — a place in a system where a claim is at once unobservable and authoritative. The ladder of light found the second: a rank on a quantity nobody can check. In every mystical case, a person stood at the top of it — a guru, a master, a medium — and in the secular case, a lodge’s Grand Master or a Forum Leader.
So the obvious objection is that these levers need a human oracle: a charlatan, a priest, a trainer. Remove the person and the slot should collapse.
It does not. The oracle can be a machine — and when it is, the slot is not merely uncheckable. It is unarguable, because there is no one to argue with.
The three best-documented instances are not science fiction. They are benefit systems and a bookkeeping program, and each ran the same experiment: an automated authority told hundreds or thousands of ordinary people that a fact about them was true, and the people could not contest it.
Horizon, in the British Post Office. Between 1999 and 2015, more than 900 subpostmasters were prosecuted — hundreds of them wrongly convicted of theft, fraud and false accounting — on the evidence of an accounting system whose faults the prosecuting organisation did not disclose.1 The figure that should stop any reader is this one: thirteen suicides.1 Convictions only began to be quashed in December 2020; the mass exoneration came by Act of Parliament in 2024; more than 2,750 further subpostmasters were found to have been affected without being convicted.1 A computer program was treated as a witness, and its output as testimony — an expert attended court to attest to the system’s accuracy, the law presumed the data correct, and the human beings were convicted on it. Post Office and Fujitsu personnel are under police investigation; this post names none of them.
Robodebt, in Australia. An automated debt-assessment system compared welfare records against averaged tax data and issued debt notices on the difference.2 It moved the burden of proof: instead of the agency verifying the debt, the individual had to prove they did not owe it — the “reversal of the onus” the Royal Commission named as the scheme’s fundamental defect.2 Around 470,000 debts were wrongly issued; the scheme was found unlawful; the settlement reached A$1.8 billion; and a Royal Commission — Australia’s highest form of inquiry — reported, with debts having been pursued against deceased people and payments taken from people contesting them.2 Its verdict was that “Robodebt was a crude and cruel mechanism, neither fair nor legal, and it made many people feel like criminals.”2 Deaths and suicides are in the record.2
The childcare benefits affair, in the Netherlands. Between 2005 and 2019, roughly 26,000 parents were wrongly accused of benefit fraud by the tax administration, and required to repay their allowances in full — in many cases tens of thousands of euros.3 The inquiries found the procedures discriminatory, falling particularly on parents with foreign backgrounds; the parliamentary inquiry concluded the affair violated fundamental principles of the rule of law; and the government resigned.34
Set the three side by side and the anatomy is identical — and it is the anatomy of the levers, with a processor where the priest was.
| Horizon | Robodebt | childcare affair | |
|---|---|---|---|
| the oracle | the accounting program | the data-matching system | the risk model |
| what it claimed | you stole | you owe | you defrauded |
| who had to disprove it | you | you | you |
| could you inspect it? | no | no | no |
| how long did it hold? | ~20 years | ~5 years | ~15 years |
| what ended it | a judgment, then a statute | a court, then a Royal Commission | a parliamentary inquiry, then a government’s fall |
Schematic — an illustration of the argument in §1–§3, not a measurement. The four properties are the unfalsifiable slot as the witch-and-the-debt post defines it.
Two columns are the whole argument. In each case the burden of proof was reversed — the machine asserted, and the human had to rebut — and in each case the assertion could not be inspected. Put those together and you have the witch-and-the-debt exactly: a claim whose grounds are closed to the one it is about, held by an authority that cannot be questioned. Citron and Pasquale named the combination a decade ago in the credit context: automated scoring is “pervasive and consequential,” and it is also “opaque and lacking oversight.”5
They also named the remedy in the paper’s own title — due process for automated predictions.56 Hold that; it is where this ends.
Here the post has to admit something, because the series’ own countermeasure is implicated.
The mechanism post ends on a specific prescription. Because the collapse cannot report itself, the fix is relocated measurement — stop trusting self-report, and measure the body, the behaviour, an external observer.7 The whole point of this blog is that measurement beats testimony.
Now look at §2 again. A machine measured. It displaced human testimony — “the computer said so” — and it produced catastrophe, because the measurement was not auditable and the measured had no standing.
The lesson is not “measurement is bad.” It is the exact next refinement, and it is worth stating as its own line:
Measurement is not a safeguard. Auditable measurement is. The difference between the two is due process.
This is the same distinction the mechanisms series draws between the state and how the state is read, one level up. A reading can be right and still be tyrannical if it cannot be checked; a measurement can be accurate and still be unjust if the measured cannot see it and cannot answer it. The psychometric instrument and the risk model are the same technology. What separates the blog’s countermeasure from the scandals is not accuracy. It is who can inspect it, and who can appeal.
Notice how each of the three ended, because it is the answer.
Not with a better belief. Not with a warning label, and not with a theory of mind about the people inside. Each ended with an audit that had power: a High Court judgment that the system’s evidence was unreliable and the organisation’s disclosure inadequate; a Royal Commission that found the scheme unlawful and named the reversal of the burden; a parliamentary inquiry that found the rule of law violated and toppled a government. And then, in each case, money and standing: compensation, repayment, exoneration by statute.123
This is the same remedy the series keeps arriving at from every direction — relocated, auditable evidence — and the same lesson the Costume drew from the registration experiment: a structure has no countermeasure except visibility with teeth. A register that found nothing was withdrawn in Britain; a system that found everything wrongly was dismantled by courts. In both, the mechanism of repair was the same: make the authority inspectable, and give the person it judges the standing to refuse it.
That is not a slogan. It is the specific, boring, contested work — appeals, disclosure, audit, standing, compensation — and it is the only thing in the record that has ever closed one of these levers.
The temptation here is enormous, and the series refuses it. “Opaque automated authority” is one step from “AI is a cult,” or “the state is a cult,” and that step is the label post’s error with a headline attached.
So, plainly: these are not cults, and this is not the mechanism. There is no collapse, no appraisal of a state as attainment, no holder standing on the far side of a person’s dissolution. A benefits system is administration, and its failures are failures of due process, remedied by courts and inquiries — which is precisely what happened.
What these systems share with the case studies is the lever, and only the lever: an authority whose grounds are closed, a rank on a quantity the ranked cannot see, a claim that must be accepted until disproved. That is why the post is here and not in the case studies. The case studies are about a state that gets steered; this is about a structure that judges. Strip the person out of the lever and the lever survives — which is the Costume’s finding, carried one step further: the position was never about the person.
The mystical cases needed a cosmology. The secular training needed a word. The machine needs neither — it needs a presumption, and the presumption is the whole of it: the system is right and you must prove it wrong.
Which is the same presumption the guru asks for, the lodge asks for, and the training sells. The levers do not require a belief, a person, or a metaphysics. They require an unfalsifiable claim with power behind it — and the only thing that has ever removed one is a process that made it falsifiable and gave someone standing to test it.
The blog’s own position is already this, and it bears repeating because the case cuts both ways. The mechanism post says: do not trust the self-report; measure. The scandals say: and do not trust the measurement; make it answer. Both are the same demand, applied at different points — the demand that a claim, whoever makes it, is answerable to someone it does not control.
A machine can hold the slot. It cannot be exempt from that.
Marking: documented — the three systems, their failures, and their remedies (cases, statutes, inquiries, settlements); the Citron & Pasquale treatment; the reversed burden of proof. Argument — that the unfalsifiable slot requires no person and no cosmology, only a presumption; that the mechanism post’s instruction to relocate the measurement is itself the technology these scandals misused, so the operative distinction is auditability with standing, not measurement; and — the boundary — that these are administrative failures governed by due process, not the collapse mechanism, and not cults. The boundary and the “measurement is not a safeguard” refinement are this post’s contribution.
On the British Post Office (Horizon) scandal — more than 900 wrongful convictions of subpostmasters between 1999 and 2015 on evidence from the faulty Horizon system, developed by Fujitsu (about 700 of the prosecutions were brought by the Post Office itself); 13 suicides; convictions quashed from December 2020; mass exoneration by the Post Office (Horizon System) Offences Act 2024; 2,750-plus further subpostmasters affected without conviction; and police investigations into Post Office and Fujitsu personnel (ongoing — hence no individual is named here; that an expert gave evidence attesting to the system’s accuracy, and that the courts worked from a presumption that computer data is correct, are both on the record). On the inquiry, note the sequence rather than a single date: its public hearings closed in December 2024, Volume 1 of the final report — human impact and redress — was published on 8 July 2025, and the remaining volumes were still pending at the time of writing. https://en.wikipedia.org/wiki/British_Post_Office_scandal · and see the Business and Trade Committee’s report, HC 1598, https://publications.parliament.uk/pa/cm5901/cmselect/cmbeis/1598/report.html↩︎
On the Robodebt scheme — the automated debt assessment comparing Centrelink records with averaged ATO income data; the reversal of the onus onto the individual and the limited human interaction involved in issuing notices; the finding that the scheme was unlawful; roughly 470,000 wrongly issued debts; debts pursued against deceased people; deaths and suicides in the record; and the A$1.8 billion settlement. Royal Commission into the Robodebt Scheme, Report (7 July 2023): “Robodebt was a crude and cruel mechanism, neither fair nor legal, and it made many people feel like criminals … It was a costly failure of public administration, in both human and economic terms” (p. xxix); on the “reversal of the onus and income averaging” as the defect the scheme’s ministers left untouched (p. 167). https://robodebt.royalcommission.gov.au/publications/report (archived: https://web.archive.org/web/2024/https://robodebt.royalcommission.gov.au/publications/report) · see also https://en.wikipedia.org/wiki/Robodebt_scheme↩︎
On the Dutch childcare benefits affair (toeslagenaffaire) — roughly 26,000 parents wrongly accused of benefit fraud between 2005 and 2019 and required to repay allowances in full; the procedures found discriminatory, falling particularly on parents with foreign backgrounds; the parliamentary inquiry’s finding that the affair violated fundamental principles of the rule of law; and the resignation of the third Rutte cabinet on 15 January 2021. [^welfare] carries the literature. https://en.wikipedia.org/wiki/Dutch_childcare_benefits_scandal↩︎
The welfare-automation literature, by name — on Robodebt: Carney, T. (2019), “Robo-debt illegality: The seven veils of failed guarantees of the rule of law?”, Alternative Law Journal 44(1): 4–10, https://doi.org/10.1177/1037969X18815913 — on how the rule-of-law institutions failed to stop an unlawful automated debt programme; and on the Netherlands: van Bekkum, M., & Zuiderveen Borgesius, F. (2021), “Digital welfare fraud detection and the Dutch SyRI judgment,” European Journal of Social Security 23(4): 323–340, https://doi.org/10.1177/13882627211031257 — on the Dutch court’s 2020 judgment against a welfare-fraud risk system, which is this post’s §5 in one case: an unauditable system, ended by a court that could inspect it. The wider frame is Alston, P. (2019), Report of the Special Rapporteur on extreme poverty and human rights: the digital welfare state and human rights, UN Doc A/74/493, https://www.ohchr.org/en/documents/thematic-reports/a74493-digital-welfare-states-and-human-rights-report-special-rapporteur. The use of these here — as the lever’s own literature — is this post’s.↩︎
Citron, D. K., & Pasquale, F. (2014). “The Scored Society: Due Process for Automated Predictions,” Washington Law Review 89(1): 1 — automated scoring as “pervasive and consequential” and also “opaque and lacking oversight”; the case for due process in automated prediction. https://digitalcommons.law.uw.edu/wlr/vol89/iss1/2↩︎
The administrative-justice literature this argument descends from — Citron, D. K. (2008), “Technological Due Process,” Washington University Law Review 85(6): 1249, https://openscholarship.wustl.edu/law_lawreview/vol85/iss6/2 — the direct predecessor of the 2014 essay and its argument that automated administrative decisions must be made reviewable; and Cobbe, J. (2019), “Administrative law and the machines of government: judicial review of automated public-sector decision-making,” Legal Studies 39(4): 636–655, https://doi.org/10.1017/lst.2019.9 — how the ordinary grounds of judicial review do and do not reach an automated decision. Neither paper is about these three systems; the reading of them as the levers is this post’s.↩︎
The countermeasure the series arrives at — the mechanism post §6’s instruction to relocate the measurement to a channel that survives, and §9’s “measure a channel that survives,” together with the Costume’s “visibility with teeth” — is the same conclusion this post reaches from the opposite direction: the fix is not a better claimant but a checkable one.↩︎