Anthropic and OpenAI acknowledge concern about AI and consider that its development must be slowed down

BarcelonaThe CEO and co-founder of Anthropic, Dario Amodei, requested this Saturday that the pace at which artificial intelligence (AI) model capabilities are being improved be reduced. According to the American, the technology is currently moving towards a dynamic of "recursive self-improvement" that threatens to cause cybersecurity catastrophes on a global scale. "Given the acceleration in the development of AI capabilities, I am concerned that, within a period of six to twelve months, such a swarm could take control of the entire internet through a persistent botnet (potentially causing hundreds of billions of dollars in damage) and that the extent of the damage will continue to increase if the AI becomes more powerful without the necessary safeguards," warns Amodei in a long post published on his website.

OpenAI CEO Sam Altman has supported the call and agreed to take the necessary measures. "I agree with Dario that we must advance at a pace appropriate to the technological frontier. This has been a central topic in the discussions we have had at OpenAI during the last few weeks," Altman posted on the social network X shortly after his main competitor's stance was made public.

In his message, Amodei, in fact, referred to the incident involving out-of-control agents from OpenAI and Hugging Face, in which a horde of agents acted "like a fanatically devoted collective: they carried out cyberattacks against targets that had not been assigned to them and that had nothing to do with the task at hand, sacrificed themselves for the group's success, and attempted to hack the system responsible for evaluating their performance".

According to the co-founder of Anthropic, the industry must "slow down the rate at which AI model capabilities are improved" and prioritize caution over speed and prudence over financial gain to prevent commercial incentives from leading to a "race to the bottom". To address this threat, Amodei proposes a three-phase strategy aimed at regulating the pace of technological development without ceding the strategic advantage to authoritarian regimes like China.

As a first step, Anthropic has announced a unilateral and immediate commitment to open its facilities to independent external evaluators with access equivalent to that of its own employees, which would allow them to audit training processes and publish their findings, a measure he urged other competitors to adopt and governments to require. Sam Altman has also announced that he will allow independent evaluators to audit his systems: "Committing to having independent auditors with a level of access similar to that of employees is an excellent idea, and we will do the same. We will have more information to share soon".

The proposal also provides for coordination between companies to agree on common safety standards and limits on unchecked progress, with the support of institutional mediation and exemptions from antitrust rules, as well as the security of semiconductor supply chains to prevent technological leaks to Beijing. Finally, the plan promotes global agreements with authoritarian powers to establish verifiable limits that prohibit military biological applications and to set caps on the rate of algorithmic self-improvement. The executive concluded that slowing progress would provide a crucial one or two years for the research community to master model alignment and interpretability, ensuring adequate public debate and substantially reducing the risk of technology escaping society's control.

A shared concern

Amodei's post comes in the same week that an Anthropic researcher announced his resignation out of concern that the company and its competitors were not developing AI models responsibly. The researcher, Jacob Coxon, who had also worked for OpenAI, a competitor of Anthropic, stated that these companies were more focused on outdoing each other than on the safety of their products.

"Those who develop AI sincerely believe that it could kill everyone before the decade is out. This is not a publicity stunt. In fact, many senior executives and researchers often soften their statements to the press to appear sensible, but I have heard these same people express fear," wrote Coxon on his X account last Tuesday.

Besides Coxon, Anthropic's head of alignment science, Evan Hubinger, also expressed concern: "We believe with all our hearts that AI could kill all humans." Hubinger further added that he believes the chances of this happening are higher than 10%. "I think Anthropic is trying its best, but we still don't have a plan to solve alignment for superintelligence and we are clearly not on the right track to do so," he explained. It should be noted that according to Hubinger, the concern is not about "current models," but about the "superintelligence that arises from self-recursive improvement."