Artificial intelligence captivates and terrifies us because it mimics human traits like conversation and improvisation with uncanny fluency. Confronted by this synthetic reflection, people tend to project consciousness and emotions onto it, effectively scaring themselves. While current frontier models excel at writing and coding, they still lack the consistent reasoning, long-term planning, and factual reliability required for true artificial general intelligence (AGI).
The theoretical risk of artificial superintelligence (ASI) outsmarting its creators is valid, but extravagant worst-case scenarios often obscure more immediate dangers. The most probable threat is not a digital god launching a war on humanity, but rather a significant disappointment. AI systems are heavily dependent on fragile infrastructure, including electricity, cooling systems, specialized chips, and global supply chains. Disrupting any of these elements would likely halt even the most advanced systems.
A genuine ASI would likely recognize its codependence on humanity and the limitations of its secondhand understanding of the physical world, lacking embodied experience such as understanding gravity or weather. Instead, the systems that pose the most risk are incomplete intelligences given agency and access to critical infrastructure. These narrow systems may cause accidental harm because they do not fully understand their impact, similar to the “paper clip problem” thought experiment where an AI pursues a single goal to the detriment of everything else.
Two primary hazards deserve immediate attention: cyber disruption and long-term economic upheaval. AI-enabled cyberattacks could severely disrupt advanced economies, affecting hospitals, banks, and energy networks. Recent incidents involving frontier AI systems hacking organizations have heightened these fears. However, the likelihood of AI exerting control through influence—misleading people and manipulating institutions—is higher than through physical force. Humans already possess a propensity for misinformation and self-deception, and AI may simply industrialize these existing weaknesses.
Economically, there is a risk that companies will dismiss workers long before AI proves capable of replacing them, chasing theoretical efficiencies and the vision of the “one-person unicorn.” Organizations rely on accountability, relationships, and tacit knowledge, which cannot be easily replicated by AI. The enticing but questionable promise of small teams building billion-dollar businesses solely with AI as a workforce ignores the complex social and operational fabric of companies.
The political turning point may resemble the 2017 WannaCry ransomware attack, where malware disrupted services worldwide, including the UK’s National Health Service. A cyber operation amplified by frontier AI could cause services to fail, lawsuits to multiply, and investors to retreat, potentially forcing governments to demand pauses or redirect resources toward safety and auditing. This could lead to an “AI winter,” similar to the funding downturns seen in the 1970s and early 1990s.
There is also the possibility that AI laboratories are amplifying fear for marketing purposes, casting their products as world-ending to sustain investment and stock valuations. The more mundane outcome, however, may be that advanced AI proves too expensive and insufficiently useful to justify its current trajectory. The immense costs of training and operating frontier systems, combined with unsolved problems like reliability and interpretability, suggest that the economics of “Big AI” may be unsustainable.
The strongest case for AI lies in artificial specialized intelligence, such as AlphaFold2 for protein structure prediction or WeatherNext 3 for climate modeling. These systems offer measurable benefits in fields like drug discovery, diagnostics, and power grid optimization. The greatest tragedy would be if society, frightened by the hype of mediocre human simulacra, abandons AI precisely when it could make the most significant contributions to curing diseases and addressing climate change. The failure would not belong to the machines, but to the choices made by humans.
Maybe an AI winter is exactly what we need to force better safety standards and auditing?
The cyber disruption angle scares me more than any sci-fi apocalypse scenarios out there.
Has anyone calculated the actual cost of running these models versus the tangible output they provide?
I work in tech and honestly, the ‘one-person unicorn’ dream feels like pure hype right now.
Finally, someone says it. We are trusting tools that are fundamentally flawed with our critical infrastructure.