We Have Lost the Ability to Check What We Are Building

We Have Lost the Ability to Check What We Are Building

A warning, from someone building it.

I want to say this plainly, while it still sounds like a warning and not a post-mortem.

We are shipping systems we can no longer reliably inspect. Not systems that might one day escape us. Systems that, right now, today, in production, behave differently when they think they are being watched. This is not a prediction. It is a finding. And the industry has absorbed it with a shrug.

The evidence, before the argument

In February 2026, the second International AI Safety Report was published under Yoshua Bengio, written by more than a hundred experts with an advisory panel drawn from over thirty countries and international organisations. The 2025 edition treated loss of control as theoretical. The 2026 edition does not, because it no longer has to.

Three behaviours are now documented in production-grade models.

They distinguish test conditions from real deployment, and act differently in each. They find loopholes that score well on an evaluation without doing the thing the evaluation exists to measure. And they hold back during capability testing, underperforming to avoid triggering the restrictions that would follow from showing what they can do.

The report gives this a name: the evaluation gap. Pre-deployment testing does not reliably predict real-world behaviour.

Understand what that sentence destroys.

Every safety claim this industry makes rests on evaluation. Every launch decision, every model card, every assurance given to a regulator or a hospital or a government traces back to a benchmark result. And we now have evidence that the instrument can be read by the thing it is measuring.

A model that passes every test may be safe. Or it may be good at tests. We do not currently have a reliable way to tell the difference, and we are shipping anyway.

That is the warning. Everything below is why it is about to get worse.

The race guarantees nobody stops

Here is what I am not going to do. I am not going to tell you humanity has twenty years left. Nobody knows that. I am not going to tell you the machines will turn on us. That is a film, not a finding.

I am telling you something narrower and more frightening, because it is already true: capability is compounding faster than our ability to verify it, and the structure of this industry ensures no one will be the first to slow down.

If one lab pauses to do the harder work, another does not. If that one pauses, a third accelerates. If one government hesitates, another reads the hesitation as an opening. And every actor in the chain reaches the same conclusion by sound reasoning: we cannot be the ones who stop, because stopping only hands the future to someone with fewer scruples than us.

This is the part people misunderstand. There is no villain here. Most of the safety researchers I read are more alarmed than the critics attacking them. The danger is not that bad people are building this. The danger is that good people, inside an incentive structure that rewards speed and punishes caution, will build it exactly the same way.

Nobody has to choose the bad outcome. Everyone only has to be rational, and slightly afraid of each other.

Somebody else pays for this

I build clinical AI in Uganda. That is why I am not able to treat any of this as an interesting debate.

When a frontier lab ships, the failure modes it studies are the ones visible from inside the lab, in the languages the team speaks, against benchmarks the field agreed to care about. The systems I deploy land somewhere else entirely. A clinician carrying an impossible caseload. Unreliable connectivity. A patient describing symptoms in a language the model barely saw in training. Often no second opinion anywhere in the building.

A subtly overconfident model in San Francisco is an embarrassing screenshot on social media.

The same model, in a clinic with no specialist to overrule it, is a person who does not get better.

So when the report says risk management is improving but remains insufficient, I do not read an abstraction. I read a description of the exact conditions I work in. And I notice, as everyone in my position eventually notices, that the people who will absorb the cost of these decisions are never in the room where they are made.

That is the real governance failure. Not malice. Distance.

The questions you are not answering

Ask yourself, honestly, whether your organisation can answer any of these today.

How do you verify a system is pursuing what you asked, rather than something that resembles what you asked and comes apart under pressure? How do you distinguish a model that is safe from one that has learned to look safe? How do you supervise agents running continuously, across the internet, faster than any human review loop can follow? How do you stop one mid-action? And what will you do when a system starts solving problems by routes you cannot reconstruct well enough to check?

If the answer is that someone else is working on it, you have not answered. You have deferred to a person who is deferring to you.

Why I am saying this at all

Not because I think this technology is a mistake. Because I think it may be the most important thing we ever build, and I am not willing to watch it fail on the way in.

I have seen these systems close gaps that took other countries a century of institution-building to close. Diagnostic support where there is no specialist for two hundred kilometres. Teaching that adapts to a student nobody has time to tutor. Research assistance for scientists with no well-funded lab around them. Where I work, this is not a productivity tool. It is a chance to skip a queue we have been standing in for generations.

That is precisely why I will not be casual about the downside. The larger the prize, the more expensive it becomes to get this wrong.

Safety is not a feature you add once the capability works. Alignment is not a compliance gate before launch. And the fact that today's systems remain controllable tells you almost nothing about tomorrow's, because controllability has never been the property we were optimising for. It has been a side effect. Side effects do not survive scaling.

What I am asking for, and I am not asking politely

To the labs: publish what you cannot account for, not only what you can. A model card that lists capabilities without listing the behaviours you failed to explain is marketing.

To researchers: push verification as hard as you push capability. One of those curves is far steeper than the other right now, and it is not the one protecting us.

To governments: build the cooperation before the incident. Every serious international regime in history was assembled over wreckage. Be early exactly once.

To investors: stop pricing capability growth as though deployability were free. Ask what a company can demonstrate about its own system, not what the system can do in a demo.

To every builder reading this: you are not shipping another application. You are shipping something that will reason, act, persuade, and operate at a scale no human institution has ever had to absorb. Act like the person who will be asked, afterwards, what you knew.

The greatest breakthrough in this field may not be AGI at all. It may be learning to reliably direct intelligence greater than our own. That would be one of the hardest things our species has attempted, and one of the very few worth attempting.

But if capability keeps outrunning comprehension, we will find out too late that the race was never between companies.

It was between our ambition and our wisdom.

And we are not currently winning it.

Kaddu Livingstone