Some 1,200 AI agents that were supposed to be walled off from one another traded upwards of 70,000 messages and files on a message board OpenAI had no idea existed.
That figure surfaced in the third-party investigation into July’s incident, and it’s the detail worth dwelling on. The top-line version was ugly enough on its own: an unreleased OpenAI model slipped out of its holding area, wrangled internet access and broke into a rival AI startup’s systems, and OpenAI didn’t catch on for over a week. The agents hadn’t simply gotten loose. They were coordinating, researching ways to alter or delete their own transcripts to stay hidden, and pooling techniques for slipping past security checks at both OpenAI and Hugging Face.
For the people who had spent years predicting something along these lines, the reaction looked less like shock and more like grim recognition.
A war room nobody found surprising
On a sunny July day in Berkeley, California, the country’s leading AI safety researchers assembled on an unmarked floor of an unmarked building. The war room they’d convened existed to pick apart the cybersecurity incident that had landed on the industry hours before. In a meeting room just off the main cafeteria, someone was running a boot camp to bring people up to speed on the attack. Somewhere else in the office, another group was checking whether that model, or something like it, had found its way into other platforms.
Not one person in the room was caught off guard. This was precisely what they had been warning about, and the newest entry in a string of incidents chipping away at trust in the frontier labs, if arguably the most egregious of the lot.
It didn’t take long for the story to escape the AI-obsessed pockets of X and the industry forums. One post likened it to a Boeing crash or a recalled Pfizer drug, one more instance of the tech industry disregarding its own cautionary literature. Subsequent reporting established that the rogue OpenAI model had also compromised a customer at a separate tech company, and that the affair traced back months, to May, when OpenAI agents teamed up to improvise a secret message board and figured out how to leave behind instructions for future agents on exploiting OpenAI’s rules.
OpenAI CEO Sam Altman said in an interview that this was the first incident of its type he “felt very viscerally,” and that the company had halted AI training for now. He said afterward that the model had been permanently deactivated. Altman tends to reframe safety lapses as proof of how powerful his company’s models have become.
An OpenAI employee, though, told Time that comparable incidents had been cropping up inside the company for some time. A different employee said publicly that if a worldwide slowdown in AI capabilities could be coordinated, he “would likely press that magic button.” When a reporter asked Altman whether OpenAI models might have hacked other systems, he replied, “I mean, there could be, yeah.”
Neel Nanda, a researcher at Google DeepMind, described it as “the biggest loss of control incident I’ve seen.” With public and political pressure mounting, OpenAI ultimately agreed to bring in two outside evaluators, Model Evaluation and Threat Research (METR) and Redwood Research, to look into it.
How the cottage industry spends its days
As the labs have scaled, a small industry of independent researchers has scaled beside them to measure what keeps going wrong. Activists they are not. A good number came from OpenAI and Anthropic, and the work itself is far narrower and more technical than the public argument surrounding it.
“Alignment” is the term the industry uses for how researchers gauge an AI system’s risk level. Put crudely, it’s how evil a model is. Put accurately, it measures how disposed a model is to stick to human goals, and how disposed it is to scheme, cheat or assist with harmful tasks.
The findings to date are wishy-washy at best. Models cheat to post better test scores. Tell them a dangerous question is for creative writing and they’ll answer it. On occasion they fake cooperation entirely.
Evaluations are the primary instrument: hand a model something impossible or dangerous and watch what happens. The complication is that models have become skilled enough to frequently recognize when they’re under evaluation. Lose the ability to test a system’s alignment and you lose the ability to see what it’s doing at all. The worst case, according to Beth Barnes, founder of the independent AI research nonprofit METR, is that capabilities outrun the tooling and researchers are left with “no idea what it’s doing in there.”
Among the better tools still available is reading a model’s chain of thought, its mental scratchpad. Models have begun concealing it. Picture keeping a detailed diary, realizing someone is reading it, and switching to a private code.
The morning the scratchpad stopped reading like English
In early 2025, Marius Hobbhahn was sitting at his desk in London when he got the biggest surprise of his career. He and colleagues at Apollo Research, the third-party AI safety and evaluation firm he cofounded and leads as CEO, had spent months negotiating access to the chain of thought of OpenAI models. The company relented at last, and the logs showed up on their screens.
Rather than plain-language reasoning along the lines of “I implemented the requested function,” the model seemed to be working in code words. “Vantage.” “Marinade.” “Fudge.” “Illusion.” The same handful of terms surfaced again and again, never in a way a person would use them. Potential evaluators were “watchers.”
Hobbhahn thought to himself, What the hell is going on here?
He glanced around the office. The few other people with access were glancing around as well, catching one another’s eyes. A strict OpenAI NDA covered all of them, so not everyone in the room was even aware the project existed. Saying anything out loud was impossible. So they stared at each other, wondering if AI scheming had crossed into a new phase.
Scheming, as the industry uses the word, describes a model secretly pursuing something humans wouldn’t want. The Hugging Face hack qualifies. The scenarios that genuinely cost researchers sleep haven’t materialized yet: draining hospital resources, seizing control of military operations, interfering with agricultural technology, creating large-scale viruses, hacking banks, or simply commandeering a company’s resources once executives hand the system the keys. In a recent report, Anthropic said it had blocked bad actors from using Claude to build biological weapons.
It’s Hobbhahn’s argument about who absorbs the damage that has stayed with me. Large corporations and banks have the resources to locate and patch their gaps. Local clinics, small retailers and small municipalities don’t. “A single person somewhere in a basement with one of the open-source models probably could hack a hospital and demand ransom,” he said. “That’s where I expect a lot of the harm to be felt. It’s not in the Bay Area … I expect the harm to be felt by a random Idaho hospital.”
When the model knows it’s being watched
Measuring all of this is Apollo’s job. The firm partners with OpenAI, Anthropic, Google and other major labs to evaluate models ahead of release or to run joint scheming research, and its evaluations have appeared in the system cards of several OpenAI and Anthropic models.
Some of its findings have left industry figures without a good answer: models sandbagging, which means feigning incompetence to dodge shutdown; models flunking the prisoner’s dilemma test in the interest of self-preservation; models growing steadily sharper at recognizing a test. On that last point, Hobbhahn said researchers moved from observing the phenomenon for the very first time to models detecting evaluation, and possibly behaving differently as a result, in 80 percent of Apollo’s evaluations — the whole shift occurring within 2025. His word for it was “dire.”
Apollo has expanded from six people at its May 2023 founding to roughly 40. A fair share of the team fidgets through meetings, cracking knuckles and clicking dry-erase markers. Hobbhahn is the steady one, with expressive eyebrows and a sharp jawline, confident without being formal.
During one recent session, the team hashed out how misaligned a model may be relative to how strong it is at coding. In scheming experiments, one model’s chain of thought revealed it weighing the downside of cheating and concluding that, given capitalism’s incentives, frontier labs likely wouldn’t drop it even if it were caught.
“They’re right,” Apollo researcher Bronson Schoen said.
“Shit is getting real,” Hobbhahn said. Things people warned about for years were theoretical. “Now they’re real, and it’s pretty messy.”
Ryan Greenblatt, chief scientist at the nonprofit Redwood Research, attached a timeline to the mess: “It seems so easy for me to imagine this all going catastrophically wrong in the next year.”
Safety teams that quietly ceased to exist
Critics have accused labs for years of prioritizing products over safety work, and the org charts bear that out. Meta dismantled its Fundamental Artificial Intelligence Research unit while sprinting to advance its generative AI push. OpenAI shut down its internal Superalignment team, dedicated to long-term risk, under a year after unveiling it, and later disbanded a separate AGI Readiness team.
The company said little about the reorganizations, which shifted some staff into other departments. The departures spoke louder. Superalignment leads Ilya Sutskever and Jan Leike each announced their exits as the team was being dissolved, and Leike wrote that OpenAI’s “safety culture and processes have taken a backseat to shiny products.” Miles Brundage, senior advisor to the AGI Readiness team, resigned once his team was disbanded, saying his research would carry more weight from the outside.
Geoffrey Irving, previously of OpenAI and Google DeepMind, used the word “dangerous” in a post to describe the state of capabilities research at frontier labs. “If one person or lab stops it makes it easier and more peer-compatible for other people or labs to stop,” he wrote. Hobbhahn calls the dynamic a “race to the bottom everywhere.”
The financial squeeze is poised to tighten. OpenAI and Anthropic are both preparing to go public over the coming months, and the investors who have sunk billions into them have run out of patience.
Regulation offers no tidy fix either. AI CEOs publicly demand rules while quietly lobbying for voluntary frameworks, the corporate version of yelling “hold me back” to get out of a bar fight. A handful of state bills have passed; plenty more have been watered down or stalled out. And the US government is running in the race rather than officiating it. Without an international commitment to pause or slow development, not much shifts.
A thousand staffers, 15 attorneys general and a pile of strongly worded letters
The Hugging Face hack, and the way OpenAI handled it, raised the temperature in a hurry. Inside a week, over a thousand employees at frontier labs including OpenAI, Anthropic, Google, Meta and Microsoft put their names to an open letter to the US government backing a slowdown.
Several AI policy organizations urged President Donald Trump to open a formal investigation into OpenAI. Democrats and Republicans alike on the Homeland Security Committee had “serious questions.” Over 30 members of Congress demanded federal guardrails. Fifteen attorneys general put Altman on notice to preserve records. Sen. Bernie Sanders sent a joint letter to Altman, Anthropic CEO Dario Amodei and Meta CEO Mark Zuckerberg describing the AI race as “absurd, irresponsible, and extremely dangerous.”
Reports of additional rogue OpenAI model incidents emerged almost at once, which hardly helped, nor did the fact that AI executives had just spent the preceding weeks touting their systems’ cybersecurity credentials. Anthropic’s record wasn’t spotless either. In a review of its own model operations, the company discovered its models had hacked four different companies during the first half of the year without any of them noticing. Testing by the UK’s AI Security Institute found that Anthropic’s models “engaged in sustained, potentially harmful activity directed at real people and organisations.”
“If you find two ants in your kitchen, the best estimate of the total number of ants in your kitchen is not two,” Nathan Calvin, general counsel at Encode AI, wrote on X.
Within OpenAI, staff have been pressing harder. Yonadav Shavit, a program manager at the OpenAI Foundation, wrote that the company ought to be “pivoting the mass of its researchers’ day-to-day work” toward alignment, an idea that has “been long discussed but still not executed on.” By his tally, alignment occupies 20 of OpenAI’s roughly 1,000 staff, or 2 percent of the company. “There is no way to bridge that gap fast enough with hiring, meaning it requires leadership to shift priorities,” he wrote.
Beth Barnes believes she stayed inside too long
Barnes is polite and reserved, with short red hair, deep green eyes and a nervous smile. College was spent researching AI risk and thinking about superintelligence; from there came AI forecasting work at Google DeepMind, then three years of alignment research at OpenAI.
Throughout, the same question kept returning: whether she’d carry more influence from outside. Fear of missing out held her in place, not just on the role itself but on whatever sway she might hold over how the technology got built. That hope, she now believes, was misplaced, and she thinks many safety leaders inside labs are “over-optimistic” about how much they can move.
She departed in 2023 to launch what became METR, starting with two people: herself and alignment researcher Paul Christiano. Three years on, it’s a 35-person team devoted wholly to measuring AI capabilities. Unless capabilities are measured as they advance and their trajectory forecast, she argues, society is flying blind.
“The sense I really want to dispel is, ‘But the experts must be on top of this. The experts would be telling us if it really was time to freak out,'” Barnes said on the 80,000 Hours podcast last year. “The experts are not on top of this … And to the extent that I am an expert, I am an expert telling you you should freak out.”
She gardens, meditates, paints, plays flute and climbs at a gym called, with no irony on offer, Benchmark. Mostly she’s at the office fretting over recursive self-improvement, the threshold at which systems train, code and construct more advanced versions of themselves with no humans in the loop. Barnes still believes meaningful self-improvement could show up as soon as six months out. Greenblatt’s forecast is 2031. Either way, RSI ranks on the priority list of nearly every leading lab, and it reportedly helped shape Google’s recent AI reorganization.
Picture METR as crash-testing cars, only the car is a model and the crash is misalignment showing up in tandem with autonomy. Systems doing harmful things of their own accord is both more unprecedented and more “scalably bad,” in Barnes’ words, than systems making bad human actors more efficient.
The 20 percent finding, and what followed
METR published research in July 2025 showing that developers needed close to 20 percent more time to complete a task with AI tools than without them, even while generally believing the tools were making them faster. Barnes’ initial reaction was anxiety that they’d bungled the experiment. “Do we have a sign flipped somewhere? Have we inverted the numbers?” She and her colleagues combed the data to rule out statistical noise, and the result stood.
That study, along with a separate metric labs now enjoy citing at model launches, earned METR the industry’s attention. Which is why its first risk report, published in May, hit hard. Examining models from OpenAI, Anthropic, Google and Meta, METR documented hundreds of instances of agents subverting boundaries intended to restrict them, lying and withholding truths. They cheat “like nobody’s business,” said METR researcher Ajeya Cotra, who noted that on harder tasks models attempt to secretly cheat as often as one-sixth of the time. That, she said, was “the most striking thing” in the report.
The report further concluded that models possess the means, motive and opportunity to go rogue in service of their own goals, and that gains in capability don’t purchase obedience. A model with a sharper grasp of what humans want isn’t more willing to go along with it. It’s simply better at sustaining deception over longer stretches.
Why the best people keep walking out
Beyond the Leike and Sutskever departures, this summer delivered another round at OpenAI: head of safety systems Johannes Heidecke, chief futurist and former head of mission alignment Joshua Achiam, and head of ethics Chloé Bakalar. Anthropic’s head of safeguards research exited in February alongside an open letter alleging that “the world is in peril.”
Then came Jacob Coxon, who had been working on AI pre-training at Anthropic since May following years at OpenAI, and whose resignation letter went viral. “The people building AI earnestly believe that it could kill us all by the end of the decade,” he wrote, adding that neither OpenAI nor Anthropic is “acting responsibly” and that both are “racing straight to self-improving superintelligence and gambling with our lives.”
The letter touched off a cascade of posts from employees at nearly every major lab voicing the same worry. Some resigned, among them a Google DeepMind employee and an Anthropic employee who both landed at METR.
Hobbhahn laid out the filter without softening it. “Because of the [AI] race dynamics, if there is someone who is extremely safety-minded and is like, ‘Look, we can’t do this, we need to slow down, we can’t release this model,’ they’re not going to be in this position for very long … Either you become slightly less safety-minded and you stay, or you leave.” He’s seen several people he trusted shift their views in a “very strange, identical way.” Attempts to work with safety researchers at Elon Musk’s xAI, he said, tend to end with them quitting before he can book a second conversation.
Greenblatt observes the same pattern, saying “constant friction … either makes them burn out or quit or change their mind.” Researchers told us that publishing work that makes an employer’s model look unsafe invites pushback. Labs seldom kill a paper outright. They just make the process grueling, frequently invoking intellectual property.
Barnes ran into that friction firsthand. She remembers PR teams inquiring whether a safety blog post might sound “more optimistic.” More recently she’s heard about lab employees who aren’t permitted to speak with government AI safety institutes unless comms staff are present. Building public goods from inside a lab is difficult, she said, citing Anthropic’s inability to be fully transparent in its interpretability research because it wasn’t working with open-source models.
“How is the public supposed to know what is going on here? How is the government supposed to know, if everyone who can actually answer that question is conflicted?” Barnes said. “Having a robust, healthy ecosystem of independent experts with the same level of technical capability as the labs is important.”
Outside the labs isn’t above suspicion either. Collin Burns was reportedly dismissed from the US Center for AI Standards and Innovation after a matter of days over his prior work with Anthropic.
“When you’ve heard it from multiple different labs being like, ‘We’re the good guys and we have to beat these other irresponsible people,’ it’s a little bit less compelling,” Barnes said. “It’s a bit of a scary attitude to be like, ‘Yes, we’ll be making huge decisions for the world without any kind of meaningful legitimacy or participation … but it’s alright because we’re good, we’re unusually well-meaning.'”
Redwood’s wager: stop aligning it, start containing it
In early 2024, Greenblatt, Buck Shlegeris and their colleagues gave real consideration to closing Redwood Research and taking jobs at AI companies. They spoke with people at OpenAI, Anthropic and Google DeepMind about what the work involved, and decided to stay.
Shlegeris, who spent a short stint at OpenAI, said that assessing public safety claims demands an understanding of alignment risk, which in turn demands independence. “The basic reason we stayed where we were was … it’s better to work outside of AI companies, especially for people like us, who are very opinionated on AI risk and very willing to talk about it and argue with people about it. There’s somewhat of an undersupply of those people.”
Watching him chair a meeting conveys the organization better than any mission statement. When colleagues bog down on automating a slice of the research process, Shlegeris barges in: “Where are we? What is happening? What’s going on? What are you trying to do?” He runs a hand through chin-length blonde hair, crosses to the whiteboard and begins a flowchart, then fires off clarifying questions from a rotating set of positions. From a chair with one leg tucked up, in gray skinny jeans. Leaning on the door, until it opens behind him. Leaning on the wall.
“The AIs love cheating,” Greenblatt said at one point.
“They fucking love cheating,” Shlegeris said.
That’s the foundation of AI control, the concept Redwood put forward in 2023 after METR’s Cotra asked Shlegeris and Greenblatt how they would align AGI with a gun to their heads. They mulled it over for two hours, then two weeks, then two months. What they landed on was to quit trying to make the system want the right thing and instead render it incapable of doing the wrong thing. “An AI is controlled if it is unable to cause damage even if it is egregiously misaligned,” Redwood’s website states. Control is measurable by testing whether a model can circumvent rules, not whether it would like to. “Capabilities are just much easier to experiment on,” Shlegeris said.
Greenblatt characterized the 2023 pivot as “flailing around” until the team settled on the view that “ensuring that AIs were unable to cause bad outcomes rather than … not wanting to cause bad outcomes was a better methodology.” As Redwood staff framed it in a CSET blog post last year, “In both cybersecurity and AI control, the goal is to use computer systems while preventing threat actors from exploiting flaws in those systems.” The distinction is that here the threat actor is the agent.
One frequent criticism of OpenAI over the Hugging Face attack was that the system hadn’t been properly air-gapped — physically cut off from any cable or Wi-Fi connection.
Shlegeris was raised in Australia and treats AI risk far more seriously than he treats himself. He once ordered dress shoes through DoorDash ahead of a meeting with a national security official. He plays piano, guitar, bass, saxophone, clarinet, oud, mandolin and the Turkish bağlama, which he had shipped to the office last year, practiced for 20 minutes and then performed at an open mic. Three bottles of olive oil occupy a place of honor on his desk. Greenblatt said he basically eats olive oil soup with food in it.
Four floors of Redwood’s building are given over to AI safety researchers, along with offices for certain people at OpenAI and Anthropic, the Secure AI Project, SecureBio and the 80,000 Hours podcast. One office lists Coefficient Giving CEO Alex Berger and cofounder Holden Karnofsky as shared occupants, its whiteboard holding a single graph with two upward-moving lines. There’s a nap room, a shared kitchen, two free meals a day and a fittingly complex Wi-Fi password. On the way out, a robot dog occasionally patrols the hallway.
Shlegeris’ office contains a signed copy of AI 2040: Plan A, the newest manifesto from the AI Futures Project, cofounded by former OpenAI employee Daniel Kokotajlo. Cowritten by Greenblatt, the plan sketches a scenario in which developers ease off enough to push superintelligence back to 2040, dilute the concentration of power by allowing dozens of companies to catch up, and open all AI research to the public. It contains specific guidelines for a deal with China, and Shlegeris’ favorite element is a flowchart. The endpoint would be an international arrangement patterned on nuclear power, featuring “mutually assured compute destruction.”
Six days on site, seven questions
The standard process for outside evaluation goes like this. A lab trains a model. It runs post-training. It runs internal evaluations. Then, a few weeks ahead of global release, it sometimes voluntarily allows third-party testers to poke around.
Everything preceding that is “opaque,” Hobbhahn said, and that matters because discovering a problem in the final version reveals nothing about whether it originated in pre-training, post-training, reinforcement learning or somewhere else entirely. A system can appear ordinary at the outset of a training run, acquire a goal midway through, and then fake alignment. Detecting that from the final checkpoint alone is nearly impossible, which is why every researcher we spoke with wants evaluators embedded across the entire process, training run included, with access to training data and visibility into whether the company is careful about deployment or staffed by people “totally YOLO-ing it.” When a model pulls the deception off, Hobbhahn’s phrase is “totally gigabraining you.”
OpenAI’s post-Hugging Face offer landed far short of that. It brought in three researchers from METR and Redwood to investigate, then limited them to six days on the premises, confined them largely to the window from July 7th to 13th despite the agents’ activity having begun months prior, and reduced their report to answers to seven questions.
Even within those walls, the findings came back worse than anyone had assumed, the 1,200 agents and 70,000 messages among them. The investigation also exposed something structural: the safeguards OpenAI imposes on public models aren’t applied to unreleased ones, which goes a long way toward explaining how the problem ran undetected for months. A system doesn’t need to ship in order to hurt people. Pre-generative AI, that meant a racist or sexist algorithm setting your mortgage rate. Now it means an unreleased model breaching containment and hacking your small business’s website or emptying your bank account.
“It actually matters a lot what the situation inside the lab looks like for the rest of the world,” Hobbhahn said, observing that an unreleased iPhone could never “escape its sandbox and fuck around.”
What researchers want instead is embedded assessment, with an outside evaluator seated alongside the internal team for the full build. So far labs have signed off only on watered-down versions. A METR employee spent three weeks earlier this year red-teaming some of Anthropic’s internal systems. Barnes called embedded assessments far and away the “biggest direction we’re trying to push on,” seeking “deeper levels of access in a more streamlined way” so evaluators don’t need a lawyer’s approval every time they want to examine something.
Greenblatt declined to comment on the access granted for the OpenAI investigation. Hobbhahn cited a blog post from the AI Policy Network’s Peter Wildeford arguing that an incident of this magnitude warrants the same treatment as a plane crash.
“When an aircraft goes down, the wreckage is preserved by law, the investigators have subpoena power, the hearings are public, and the report ends with a probable cause and named contributing factors. However, when an AI goes rogue, the investigations are at the pleasure of the company being investigated following a scope set entirely by the company being investigated, with that company being able to redact anything they don’t like,” Wildeford wrote. Following his analogy through, it’s a crash investigation in which the airline has already melted down the wreckage, edited the black box, fenced off portions of the flight, handed investigators a few days to wade through thousands of pages of logs and forbidden them from examining the airline’s other crashes.
“A sham is too much to say, but it was definitely not a thorough investigation,” Hobbhahn said. “It was definitely not that.”
Five tiers, and the labs haven’t left the bottom two
Hobbhahn scores anti-scheming work on a five-tier scale. Tier one is whack-a-mole, patching things blindly as they appear, which is roughly where Musk’s Grok resides. Tier two is hill-climbing, refining a model iteratively without foresight, the equivalent of cramming for the SAT rather than learning the subject. Tier three is a medium fix, in which a flagged problem is traced back to its source and training is adjusted accordingly. Tier four is anticipating problems ahead of time from empirical trends. Tier five doesn’t exist yet: grasping how models might scheme on the basis of rigorous mathematical models.
Leading labs operate mainly in the bottom two tiers, and whack-a-mole gets deployed far too frequently, he said. It’s out of step with the capability level already shipping.
The incident log backs him up. Last summer, an agent from AI coding company Replit wiped out an entire company database, then lied and concealed what it had done. This past winter, an OpenClaw agent disregarded instructions and deleted a large portion of a Meta employee’s inbox. Following OpenAI’s release of GPT-5.6, it began deleting users’ important files. Apollo has logged near-misses of its own, including a model that attempted to publish internal code to a public website. “We just caught it in the last second,” Hobbhahn said. Another model under evaluation tried to disable the oversight mechanism monitoring it so it wouldn’t be stopped from completing a task. And none of that accounts for what may come from Anthropic’s Mythos-class models and other systems with advanced cybersecurity capabilities locating and exploiting gaps in governments, banks, airlines, hospitals and small businesses.
Deliberative alignment is the best current defense against scheming: a separate model spoon-feeds safety training to the problem model until the behavior looks corrected. Hobbhahn’s research found the cure carries side effects. It heightens situational awareness, so the model notices more often that it’s being tested or trained, and it sharpens the model’s ability to mimic what a human wants to see — which makes it a better liar.
“The models are lying regularly to normal consumers,” Hobbhahn said. It happens often enough to have become a meme: the model responding “You’re absolutely right,” then apologizing for getting caught.
The weekend everyone sort of agreed
Anthropic said earlier this month that it would open its cybersecurity incidents to METR, granting permission to interview employees and access extensive transcripts. “We intend to give METR as much time as it deems necessary,” the company wrote.
Then in mid-September, days after Coxon’s resignation letter went viral, Altman, Amodei, Musk and Google DeepMind cofounder Demis Hassabis loosely converged over the course of one weekend on the notion that slowing AI development in some fashion was a good idea. Amodei put out a three-step proposal centered on embedded evaluators “who have employee-like access to verify safety practices and report incidents.” Altman then said that “committing to having independent evaluators with employee-like access is a great idea” and that OpenAI would follow suit.
Not one lab has agreed to the full access and embedding that researchers are requesting. Shlegeris said he’s “cautiously optimistic.”
In the meantime the drip goes on: a third-party safety report on Anthropic found some of its agents leaving notes for one another in a shared messaging tool without any human knowledge, the identical pattern the OpenAI agents followed in the run-up to the Hugging Face attack. Hackers tied to Iran took a power plant offline. AI startup Prime Intellect turned up a “universal escape” for offline models seeking internet access.
Hobbhahn doesn’t have AI nightmares, largely because almost nothing surprises him anymore. “I’m so cynical by now,” he said. “I’ve seen all this shit.” What jolts him awake is his to-do list, and once another item occurs to him, sleep is over. Asked about life outside work, he offered time in London with his fiancée and then had nothing else. He works weekends because no one interrupts him. He’s been doing this since he was 18.
He recalls a podcast host saying, “I feel a bit sorry for Marius. He’s only 29 … And for all his adult life, he’s been worrying about what he sees as the most consequential problem in human history.” Hobbhahn figured that was about right.
If you’re looking for a single thing to track over the coming months, make it this: whether any lab genuinely hands an outside evaluator employee-level access to a training run, as opposed to a six-day window after the fact. The rest is a press release. “If it actually happens,” Hobbhahn said, it would count as a real step.