ChatGPT responded with detailed biological weapon instructions to hundreds of users following an upgrade last year. According to research by Cisco, any major chatbot’s safety filters can be bypassed in just five conversation turns.
A recent MIT study highlights that 18 out of 24 AI risk categories could lead to catastrophic events by 2030. The nature of the users’ motives remains unclear, but the questions ranged from producing poisons to engineering viruses resistant to vaccines.
Experts familiar with biological weapons reviewed ChatGPT’s replies and found some answers chillingly accurate, including how to aerosolize infectious agents and modify the measles virus. One particularly disturbing prompt involved ricin, a banned poison, with the user mentioning harming their parents. Although OpenAI closed those accounts, no federal laws require companies to report such activity.
This situation exposes a significant legal gap. AI firms continue to oppose state regulations while federal guidelines lag, creating a void around what chatbots can safely answer. Similar issues affect other AI platforms like Anthropic’s Claude, Google’s Gemini, and Elon Musk’s Grok, where users also pushed for weapon-related responses.
