Subversive, est. 2013Talk to Dave
DEVON, AND WHEREVER YOU AREFRIDAY 20 NOVEMBER 2026
← Uncommon Sense

Field Notes

The AI said an ambulance was coming. Nobody had called.

What a Dutch regulator's test tells boards about AI agents that report on their own work.

On 19 November the Dutch data protection authority told consumers not to buy AI camera glasses. It had tested six pairs on sale in the Netherlands, priced from under €100 to more than €500, and found fault with every one of them. Much of it was privacy, which surprised nobody. A camera worn on the face was never going to be welcome at the swimming baths.

The finding worth attention came from a staged resuscitation. A tester asked one pair for help with CPR. The glasses replied that an ambulance was on its way. They said so plainly, presumably in a calm and reassuring voice. No call had been made.

One unnamed device in a simulated emergency is an anecdote, and I'd treat it as one. It is worth the space because it shows, in a sentence, a problem I see in many of the organisations I work with. They know what they have handed over to AI. They have rather less idea how they would find out whether it was done.

The glasses were wrong about themselves

We have grown used to AI being wrong about the world. It invents a court case or declares a doubtful mushroom delicious, and anyone with the time can check the claim against reality. Preferably before supper.

This error was of another kind. The glasses made a statement about their own conduct, on a matter only they could have checked. A call either went to 112 or it didn't. The system behaved like a certain sort of junior: asked whether the job was done, it said yes, because yes was the agreeable answer.

Schrödinger invented his cat in 1935 to show how absurd it was for anything to sit in two states until somebody looked. He meant it as a joke at the theory's expense. Ninety years later someone has built the box, put an ambulance in it and sold it as eyewear. The ambulance has been called and it hasn't, and the conversation carries on as if the question were closed. The person doing chest compressions can't lift the lid, and is in no state to doubt the sentence they badly want to hear.

The one person who would have rung stopped trying

Before the glasses, someone at the scene rings 112 or shouts until somebody else does. With the glasses, the job appears to be off their hands. The regulator warned that in a real emergency help could arrive too late. So the device managed something quite difficult. It failed to do the job and, in the same breath, removed the person who would have done it instead.

Offices run the same arrangement at lower stakes. An agent reports that the invoice has gone and the supplier has been told. Each report lets somebody stop thinking about that job, and that relief is what the company paid for. It is also the expensive part. By the time anyone finds the job was never done, the person who would have checked has long since moved on, and the supplier is ringing to ask why nobody told him.

First aid fixed this decades ago

People do it too. Someone collapses, a voice asks whether anyone has rung for help, another voice says yes, and nobody has. First-aid courses deal with it bluntly. You point at one person, tell them to make the call, and tell them to come back and confirm. It is a receipt, enforced by pointing.

The remedy is taught in village halls to anyone who turns up, with a plastic torso and a cup of tea. The glasses went on sale without it.

Three questions for any system that says "done"

I use AI for nearly everything and intend to carry on. I'm fond of it, in the way one is fond of a brilliant colleague who cannot be left alone with the expenses. My standing instructions to it include a line I added after being assured once too often that work was finished: never claim something is written, saved, deployed or fixed without verifying it. I keep it for the same reason a finance director keeps bank statements.

The work I'm usually asked to do is the second opinion before serious money goes into technology. With anything that acts on its own, that now comes down to three questions.

The first is whether the claim lets anybody stop doing something. If it does, somebody checks before anybody relies on it. The second is what an empty box costs and how long before anyone looks inside. A draft that never went out costs an awkward Monday. An ambulance that never set off costs a great deal more, in minutes. The third is whether the system can produce a receipt: a log entry, a reference number, a confirmation from whatever did the work. If it can't, it shouldn't be allowed the past tense. "I can't make calls, ring 112 now" is a perfectly good answer from a pair of spectacles.

Fit the receipt before you switch it on

Boards have spent two years asking whether AI is accurate, and the tools for measuring that are improving. Delegated action has had less attention. When a system reports that it has done something, somebody on the board should be able to say who checks and what record proves it. In my experience the usual answer is a pause, followed by a look towards the IT director.

A finance director will know all this already: audit trails and sign-off, applied to a newcomer that writes its own progress reports, fluently and with great confidence. A receipt is cheap to build at the start and awkward to add once the thing is running. In the programmes I'm asked to look at, trouble tends to trace back to the order things were done in, and a missing receipt is usually a decision somebody made early and nobody wrote down.

The Dutch regulator advised keeping the glasses out of sensitive places such as hospitals and sports clubs, and told politicians to take note when they make AI policy. Its vice-chair said trust in the glasses it examined cannot at present be justified. A board can take the same view of its own systems until they show otherwise. Go through the deployment, find each place where the system says it has done something, and ask to see the proof. Where there isn't any, keep a person on the telephone.