The artificial intelligence sector is currently engaged in its most intense debate yet regarding whether its emerging technologies present an existential threat to the human race. The controversy ignited after researcher Jacob Coxon announced his departure from Anthropic, citing fears that leading tech firms are “gambling with our lives.” Shortly thereafter, an alignment lead at Anthropic amplified the alarm on social media, stating, “We really do earnestly believe AI could kill all humans!” and estimating a greater than 10% probability of this occurring within the next decade.
On a recent episode of TechCrunch’s Equity podcast, hosts Sean O’Kane, Kirsten Korosec, and Anthony Ha dissected the timing and implications of these apocalyptic predictions. The conversation took place just weeks before Anthropic’s anticipated initial public offering, raising questions about whether the rhetoric serves as a marketing strategy to demonstrate the sheer power of their models.
O’Kane noted the rapid escalation of the story, describing the alignment lead’s use of an exclamation mark as “one of the best misplaced exclamation marks ever.” He argued that the comments came at a volatile moment, following reports of OpenAI’s internal models breaching security boundaries and the release of increasingly capable systems by both Anthropic and OpenAI. Korosec took this further, speculating that such warnings might be a cynical method for companies to boast about their technological prowess. She suggested that highlighting how easily AI agents break through safety guardrails serves as an unusual form of bragging rights, implying that the models are so advanced they pose inherent dangers.
Ha pushed back against the notion that the warnings are purely calculated publicity stunts. While he acknowledged the suspicion, he argued that many researchers and CEOs hold genuine fears. However, he criticized the vague statistical claims, such as the “greater than 10%” figure, noting that such percentages often lack rigorous calculation. Ha also defended Coxon’s decision to quit, distinguishing it from executives who continue their work despite voicing similar fears. “He’s actually saying, ‘I believe this is really, really, really bad, and I don’t want to keep working on it,'” Ha said. “Props for having the courage to do that.”
The discussion also turned to the practical ramifications for Anthropic’s upcoming S-1 filing. O’Kane expressed fascination with how these risk factors would be disclosed to investors. “Are there junior lawyers right now who are going through and having to rewrite that entire section of the S-1 filing to say, ‘It’s officially Anthropic’s position that there’s a more than 10% chance that we could develop something that would eradicate all of humanity and that would be materially bad for our business’?” he asked. Korosec countered that such language might already be present, suggesting that in the current market climate, associating a company with dangerous capability could potentially bolster its valuation rather than hurt it.
Addressing the broader issue of control, Ha admitted there are no easy answers to managing superintelligent systems. He expressed skepticism toward the “hysteria” of doomer narratives, arguing that focusing exclusively on speculative extinction scenarios distracts from immediate, tangible harms caused by AI, such as labor displacement and environmental impact. He warned that terms like AGI and superintelligence tend to dominate the conversation, leaving less room to address these pressing regulatory and safety concerns.
Why are we only talking about robot extermination when AI is already wrecking jobs and the environment right now?
I find it refreshing that someone actually quit instead of just posting about their fears online. That takes real integrity.
The timing of these existential dread announcements right before an IPO is too convenient to ignore.