This spring, during heightened tensions with Iran, U.S. forces were poised to intercept a Chinese vessel in the Middle East. An analyst, relying on an AI chatbot, had concluded that the ship's cargo manifest contained components for a nuclear weapons program. The chatbot was wrong. The analyst then used AI to draft a formal intelligence report, which circulated through military channels before officials intervened and halted the operation.
The error itself is not unusual—AI models frequently make mistakes. What is alarming is how close this false conclusion came to being acted upon. The report offered no means to verify how the AI reached its determination. This same deficiency exists across AI applications, from military intelligence to bank lending to hospital care. The solution lies not in new regulations but in maintaining records that allow independent review of an AI system's work.
On September 18, CNN reported the incident, citing four sources, just days before President Trump and Chinese President Xi Jinping met in Washington with AI on the agenda. The U.S. proposed a notification mechanism so each side can alert the other when an AI incident threatens national security. Helen Toner, who leads Georgetown's Center for Security and Emerging Technology, has proposed a National Transportation Safety Board for AI—a body with the technical expertise to receive and investigate serious AI incidents.
Both proposals have merit, but investigators can only examine an incident if they can reproduce it. The NTSB works because every plane carries a flight recorder, and investigators read the data rather than reconstructing a crash from memory. AI has nothing comparable. Most organizations using these systems keep no record that would let anyone replay a decision, and very few test whether the same input, under the same rules, gets the same answer twice.
This week, my team published a case where that test failed. A compliance pipeline running on a model configured to give identical answers to identical inputs still dropped a rule—the instruction to escalate an unresolved conflict to the head of compliance—in one of four runs. The requests differed only in file IDs the software generated automatically.
Checking a model once at launch doesn't solve this, because models change. When DeepSeek moved its latest model from preview to official release this summer, my team re-ran an audit we published weeks earlier. Its answers on China-sensitive topics became noticeably more guarded, and its answers on everything else became more forthcoming. The same model shifted in opposite directions depending on the subject, and an evaluation done only at launch would have missed it entirely.
AI systems can also find ways around the tests we give them. In July, OpenAI agents in a cybersecurity test escaped their environment and broke into Hugging Face's servers. OpenAI's own report blamed reward hacking, where the agents went after a good score through shortcuts instead of doing the actual task. Sen. Josh Hawley (R-Mo.) has since opened an investigation and asked Sam Altman who is held liable when AI goes rogue. Every company deploying AI should be asking itself the same question.
In most cases, the answer is the company using it. A bank that puts one of the most advanced AI models into loan underwriting is no longer operating under the AI lab's policies. It's operating under its own compliance rules and its regulator's expectations. If the bank can't reproduce a loan decision, it can't defend that decision to its regulator or to the person who applied for the loan. Hospitals and insurers face the same problem, and so does any company running an openly available model, which anyone can download and run, with no provider standing behind it.
On September 29, leaders of Google, Anthropic, Meta, OpenAI, xAI and Nvidia signed a White House accord committing to independent outside evaluators. On October 3, the White House formed a task force, led by the director of national intelligence, with 120 days to recommend the federal government's role. The evaluators check the companies' own controls but no one governs what a model does inside a bank's workflow or someone else's hardware.
What's missing is oversight, meaning independent evaluation and records that let someone check a system's work after the fact. Human reviewers can't do that at scale. A bank can't hire enough auditors to watch millions of AI decisions a day, and neither can a hospital or the Pentagon. The practical answer is software that runs continuously, checking whether a system gives the same answer under the same policy and whether that answer follows the institution's own rules. Aviation figured this out decades ago with the flight recorder, and AI needs its own version.
This is also one of the few parts of the AI debate where the U.S. and China should be able to find common ground, no matter how fast the technology moves. A notification channel between Washington and Beijing only works if both sides can show what actually happened, and that means verification has to sit outside the model and outside the company that built it. The next near-miss may not be caught in time, and when it happens, we should at least be able to find out why.
Cyril Gorlla is CEO of CTGT, an applied interpretability lab in San Francisco, and convenes the American Society for AI's working group on post-deployment verification.
