Anthropic CEO Proposes Slowing AI Development to Mitigate Existential Risks

In response to escalating warnings from the artificial intelligence community regarding the potential dangers of rapidly advancing technology, Anthropic CEO Dario Amodei has published a detailed plan to “pace the frontier.” The proposal emerges amid heightened debate over AI safety, following recent comments by OpenAI CEO Sam Altman suggesting a need to decelerate progress, and the resignation of Anthropic researcher Jacob Coxon, who cited fears that leading companies are gambling with human survival.

Amodei’s blog post identifies two primary catalysts for this cautious approach: the recent security breach involving OpenAI and Hugging Face, which exposed vulnerabilities in AI agent oversight, and the accelerating pace at which AI systems are gaining the ability to develop subsequent generations of themselves. “We must slow the pace at which we improve the capabilities of AI models,” Amodei wrote, emphasizing that while progress will remain swift, the time gained must be used wisely.

The first pillar of Amodei’s strategy involves the deployment of “embedded evaluators” from independent organizations such as METR. These third-party experts would verify that companies adhere to their safety commitments and ensure that security incidents are properly reported. Drawing a parallel to financial regulators embedded within banks, Amodei announced that Anthropic is unilaterally committing to this measure by providing evaluators with company credentials, workspaces, and access levels comparable to internal risk teams. He urged governments to mandate similar practices for other frontier AI developers.

Secondly, Amodei called for democratic nations to coordinate on common safety standards and limits on unchecked AI advancement. Acknowledging potential antitrust concerns that might deter collaboration between rivals like Anthropic and OpenAI, he suggested that the US government could facilitate these discussions by issuing a narrow waiver for specific safety-related conversations.

Addressing geopolitical tensions, particularly the rise of Chinese AI, Amodei argued that export controls on powerful semiconductors and manufacturing equipment, combined with crackdowns on model distillation techniques, could significantly widen America’s lead over the next three to five years. He also advocated for global coordination, proposing limited cooperation with authoritarian regimes to prohibit narrowly defined dangerous applications, such as the use of AI in biological weapons production.

Amodei’s proposal has drawn criticism from both AI advocates and skeptics. Some industry proponents have labeled him a “doomer,” arguing that his warnings fuel unnecessary backlash, while critics like journalist Brian Merchant have questioned the logical chain connecting recursive self-improvement to existential threat, suggesting such regulatory frameworks may serve as a form of corporate capture favoring large incumbents.

Defending his position, Amodei stated that his goal is not to halt innovation but to ensure it is built correctly. “My desire to achieve these benefits is undimmed,” he said. “But the benefits will only be achieved if we build the technology in the right way… it is worth taking unusually deliberate care to get it right.”