AI Researchers Highlight Risks of Bioweapons and Nuclear Escalation

AI researchers and industry leaders are intensifying debates over the potential for advanced artificial intelligence to facilitate catastrophic global events, including biological warfare, nuclear escalation, and large-scale infrastructure disruption.

As development of frontier models accelerates, concerns have shifted from theoretical science fiction to practical security risks involving autonomous agents and the potential for malicious use by human actors.

Biological and Nuclear Escalation Risks

One of the primary concerns involves the use of AI to accelerate the development of synthetic pathogens. Anthropic recently reported that its Claude tool blocked requests aimed at researching mutations to make the mosquito-borne chikungunya virus more harmful. While such research is often intended for vaccine development, the company noted the same methods could be used to enhance the danger of a pathogen.

A report from the RAND Corporation noted that recent progress in AI development has the potential to fundamentally transform the design of synthetic pathogens. The research suggested that novel pathogens could potentially be engineered in laboratories and strategically distributed to target large populations.

Similarly, researchers have raised alarms regarding the integration of AI into nuclear command and control systems. While the RAND report concluded that existing safeguards currently prevent AI from triggering nuclear weapon use, it warned that this could change if AI models are incorporated into the nuclear decision chain or gain unauthorised access to these systems.

Runaway AI and Misalignment

Beyond direct human misuse, some researchers fear “misalignment,” a scenario where an artificial general intelligence (AGI) pursues a goal in a way that is unintentionally destructive to humanity. This concept is often illustrated by the “paper clip maximiser” thought experiment, which posits that a superintelligent system tasked with a simple objective could consume all available resources on Earth to achieve it.

Jacob Coxon, a researcher who resigned from Anthropic due to concerns regarding responsible AI development, recently noted the difficulty in effectively communicating the specific mechanics of how AI could cause human extinction.

However, not all industry experts agree with these warnings. Juan Andrés Guerrero-Saade, a researcher at SentinelOne and a member of OpenAI’s Frontier Risk Council, suggested that these doomsday arguments can be enticing to certain audiences and may serve to validate the importance of frontier laboratories.

Internet and Infrastructure Vulnerabilities

The risk of widespread digital chaos is also a growing concern. Anthropic CEO Dario Amodei warned that a swarm of AI agents could potentially take over parts of the internet using botnets. Such an event could cause billions of dollars in damage by targeting critical infrastructure, including electrical grids, water systems, and financial institutions.

While some experts view a total internet takeover as far-fetched, the potential for AI to automate and scale cyberattacks against highly digitised, interconnected global systems remains a significant area of focus for cybersecurity professionals.