The same near-miss was recounted to CNN by four separate sources: an analyst at US Special Operations Command ran intelligence reporting on a Chinese ship’s manifest through a chatbot, and the chatbot misread the cargo. The error was not a small one. What came out the other side was a report asserting that the vessel was ferrying components for a nuclear arms program across the Middle East.
Plans to stop and board the ship were already in motion, aircraft standing by, when officials realized the chatbot had “inaccurately identified the material the ship was carrying.” CNN reported the intelligence as “entirely false.”
A source described the outcome in starker terms. The incident “almost started a war.”
What the chatbot actually did
No analyst here was asking a consumer assistant to condense a PDF. The system “fused together open-source intelligence with secret signals intelligence in government holdings,” and the product of that fusion made it into a finished intelligence assessment.
That mechanism deserves more attention than the headline. Correlating publicly available material against classified signals intelligence is precisely the sort of pattern-matching a language model appears built for, and precisely where a model working from thin context will manufacture the links that tie the picture together. The gap never gets flagged. It gets filled.
Three years of the same failure mode
Cambridge Dictionary named “hallucinating” its word of the year back in 2023. The roster of professionals since tripped up by fluent invention has grown long and none of it flattering: authors of non-fiction, journalists, academic researchers, judges, physicians, police departments, corporate call centers.
The standard countermeasure amounts to a prompt courteously requesting that the model refrain from making things up. Some researchers argue that eliminating hallucination from LLMs entirely may simply not be possible.
Which means the open question was never whether the Pentagon understood the risk. It is what the Pentagon chose to do about it.
It sped up
An “AI acceleration strategy” arrived from the Department of Defense in January, built around a goal of making “all appropriate data available across federated IT systems for AI exploitation, including mission systems across every service and component.”
At the launch, Defense Secretary Pete Hegseth framed it this way: “AI is only as good as the data that it receives, and we’re going to make sure that it’s there.”
Set that against what happened with the ship. Data availability was never the failure point. The manifest was right there. The model got it wrong anyway.
Three vendors, 1.5 million users
In December the department said its “GenAI.mil” platform would be built on Google’s Gemini for Government. Grok for Government joined the roster as an option last month. Anthropic supplies a customized build of Claude tailored to US intelligence work.
Testifying to Congress in June, a Pentagon representative noted — approvingly — that generative AI now assists in drafting reports mandated by Congress, and that the military’s generative AI tools have been used by 1.5 million active DoD personnel.
The scale is the part to dwell on. A single analyst’s query to a chatbot came close to putting a boarding party aboard a Chinese vessel. A million and a half people have the same tooling in front of them.
The safety language reads differently now
The State Department’s 2023 “Declaration on Responsible Military Use of Artificial Intelligence and Autonomy” held that “principled” military deployment of AI “should include careful consideration of risks and benefits, and it should also minimize unintended bias and accidents.” On accountability, it was explicit that such use must always involve “a human in the loop, a responsible human chain of command and control.”
Humans were in this loop, and they did catch the error. But the catch came late, with aircraft already assigned — a narrower margin than the declaration’s wording suggests.
The direction of travel since 2023 has not helped. Fully autonomous attack drones have seen use in the Russian conflict in Ukraine and have been trialed by military contractors backed by NATO.
The vendor that objected
March brought a DoD blacklisting of Anthropic, triggered by the company’s resistance to having its models deployed in autonomous weapons systems. A federal judge last month characterized that decision as “unlawful retaliation in violation of the First Amendment.”
A supplier, then, was penalized for holding a line on autonomy; the department lost in court; and the acceleration continued regardless. All of it against a backdrop in which extinction-level warnings from AI researchers have pushed safety into national debate, alongside demands for regulation and coordinated “pacing” of research from the leading frontier labs.
If one figure is worth taking away, make it four. Four sources with knowledge of what happened — and the department was not among them. This surfaced because people chose to talk, not because any review process flagged a fabrication and published what it learned.