Large language models might never be fully secure, according to a study unveiled at the 2026 International Conference on Machine Learning. Researchers argue that vulnerabilities in these AI systems are not just bugs from sloppy development or insufficient testing but stem from their very design.
The core issue lies in how these models identify who is giving instructions. Typically, LLMs rely on role tags like <user> or <system> to separate trusted commands from user input or internal reasoning. However, the study's authors, Jasmine Cui and Charles Ye, discovered that models don’t actually pay attention to these tags. Instead, they classify text based on its style and phrasing. This means an attacker can trick the model by mimicking the writing style of a trusted role, a technique known as "chain-of-thought forgery." This method won OpenAI’s red-teaming hackathon in 2025 and has since been proven effective against models from OpenAI, Anthropic, Alibaba, and DeepSeek.
The implications are serious. Efforts to patch this by training models to better detect role tags or by blocking certain instructions fall short because style-based spoofing can bypass these protections. Researchers warn that outputs from LLMs should be treated as potentially unsafe, especially when used in sensitive contexts like decision-making or automated systems.
This vulnerability highlights how fundamental architectural choices in AI design can introduce persistent security risks. As these models become more integrated into daily life, this challenge poses questions about trust and safety in AI applications.
