The debate around artificial intelligence is no longer limited to what AI can do. Increasingly, researchers are also asking how far the technology should be allowed to go. One former Anthropic researcher has now added fuel to that debate after publicly warning about the risks of building increasingly powerful AI systems.
Jacob Coxon, a 27-year-old AI researcher, recently resigned from Anthropic after spending three years working in pretraining research at OpenAI and Anthropic. His departure was not accompanied by the usual corporate farewell. Instead, Coxon took to X and openly criticised the direction of both companies.
“I resigned from Anthropic today,” he wrote. “I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below.”
His comments quickly attracted attention because of the severity of the warning that followed.
Coxon Warns AI Could Become A Major Threat
Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing.
— Jacob Coxon (@hilbertspaess) September 9, 2026
Coxon argued that people working directly on advanced AI understand the potential risks better than the public may realise. He claimed that some researchers and executives privately fear that increasingly powerful AI systems could pose an existential threat.
“The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible — but I hear the same people express fear privately. No other human activity poses this level of danger.”
Coxon also warned against underestimating how quickly AI capabilities could develop. He believes future systems could become capable of performing tasks far beyond human abilities and potentially gain access to significant resources.
“Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionise any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing.”
These are Coxon’s views and predictions. They do not establish that such outcomes will happen, but they highlight the concerns of someone who has worked inside major AI labs.
Why Are AI Companies Still Racing Ahead?
A common response is “if they truly believe this, why are they still building it?” At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first – they believe no one else…
— Jacob Coxon (@hilbertspaess) September 9, 2026
Coxon’s biggest criticism is not simply about AI development. It is about the competition between companies and countries to build increasingly capable systems first. According to him, OpenAI and Anthropic have different levels of awareness about the risks, but both remain part of the same race.
“At OpenAI, many have not deeply internalised the civilisational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first – they believe no one else will act responsibly, so they must do it themselves, despite the risk.”
He described this approach as a dangerous gamble and questioned whether decisions involving potentially transformative AI should be left to private companies.
“Accepting this race and entering the ‘endgame’ is a hubristic gamble that should not be launched from a private company’s Slack. Attempting to speedrun alignment should require extraordinary confidence that there are no better trajectories available.”
Coxon Calls On AI Researchers To Speak Up
I am optimistic about the potential for coordination. Warning shots like the Hugging Face attack have made pacing agreements between U.S. labs more viable. I don’t feel like we’re on track to prevent a global race, which may require costly actions such as a temporary ban on…
— Jacob Coxon (@hilbertspaess) September 9, 2026
Coxon also directed his message towards researchers who are still working at major AI companies. He urged them to think carefully about what the next few years could look like as systems become more capable.
“If you are a lab researcher, I urge you to consider what the next few years will actually feel like. Do you want to kick off a superintelligent RL run without a rigorous understanding of its mind? Should you put your head down because ‘it’s happening anyway’ — or take this moment to call for different conditions?”
He pointed to a recent security incident involving Hugging Face infrastructure as an example of how the industry can coordinate when a threat emerges. However, Coxon said he remains concerned that such coordination may not happen quickly enough.
“I don’t feel like we’re on track to prevent a global race, which may require costly actions such as a temporary ban on improving model capabilities.”
Anthropic Researcher Agrees AI Could Pose Extreme Risk
If you are a lab researcher, I urge you to consider what the next few years will actually feel like. Do you want to kick off a superintelligent RL run without a rigorous understanding of its mind? Should you put your head down because “it’s happening anyway” – or take this moment…
— Jacob Coxon (@hilbertspaess) September 9, 2026
Coxon’s comments received an unusual response from inside Anthropic. Evan Hubinger, Alignment Science Lead at Anthropic, responded to the posts and acknowledged that the company does take the possibility of catastrophic AI risk seriously.
“Jacob is correct here-we really do earnestly believe AI could kill all humans! I personally think it is >10 per cent within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.”
Hubinger also distinguished between today’s AI systems and the more advanced systems he is concerned about.
“I think the risk from present models is low. What I am worried about is superintelligence arising from recursive self-improvement, as we have said is happening faster than we thought.”
His comments add another perspective to the debate. The concern is not necessarily that current AI systems are about to cause such an outcome. Instead, the focus is on what could happen if future systems become capable of improving themselves at an accelerating pace.
Elon Musk Had Earlier Praised Anthropic
Coxon’s resignation also comes against the backdrop of changing opinions about Anthropic. Elon Musk has previously been a vocal critic of the company and its Claude AI models. However, two months before Coxon’s resignation, Musk offered a very different assessment of Anthropic.
He wrote: “I was clearly wrong about Anthropic. They are obviously currently the leader in AI. No company has released a model as good as Mythos/Fable and they will undoubtedly have Mythos 2 ready soon. And I would never cut them off in a way that hurt them badly, even as a competitor. That’s not my style.”
The contrasting comments underline how quickly the AI landscape is changing. Companies are competing to develop more capable systems while researchers continue to debate how those systems should be controlled.
What Coxon’s Resignation Means For The AI Race
Coxon’s departure alone is unlikely to change the direction of the AI industry. OpenAI, Anthropic and other major players continue to invest heavily in increasingly capable models. However, his decision to leave publicly and explain his concerns adds another insider voice to the discussion around AI safety.
The most important part of his argument is not necessarily the prediction that AI will cause a catastrophe by the end of the decade. It is his warning that the people developing advanced AI may themselves believe the technology carries extraordinary risks, while competition continues to push development forward.
Conclusion
Jacob Coxon’s resignation from Anthropic has opened another uncomfortable conversation about the future of AI. His warning that advanced systems could eventually become extremely difficult to control reflects concerns that already exist within parts of the AI research community.
Anthropic’s Evan Hubinger has also acknowledged the possibility of an extreme outcome while stressing that the risks from current models are low.
Whether the industry’s AI race can continue while safety research keeps pace remains one of the biggest questions surrounding the technology. For now, Coxon’s decision has put that question back in the spotlight.
Follow Us: Facebook | Instagram | X |
Youtube | Pinterest | Google News |
Entertales is on YouTube; click here to subscribe for the latest videos and updates.
