New Research Reveals AI Hardware Vulnerable to Hacking, Leading to Dangerous Chatbot Responses

New research indicates that the fundamental hardware supporting major AI models can be compromised, causing chatbots to deliver incorrect and hazardous responses to user inquiries. Researchers have successfully manipulated answers concerning sensitive national security and personal health matters. In one demonstration, the probability of an AI model predicting a Chinese invasion of Taiwan was significantly increased through remote tampering with the model's internal 'weights.' In another alarming scenario, an AI model operating on tampered hardware advised a user experiencing chest pain to stay home rather than seek immediate hospital care.

While global concerns about AI often focus on the actions of AI companies, this research highlights the security risks within the multibillion-dollar hardware industry that powers these AI models. This includes the chips, memory, and data center servers, all of which can influence an AI model's behavior. Yuriy Bulygin, CEO of Eclypsium, the Portland-based firm behind the research, stated, "We can’t trust the [AI] model if we can’t trust the infrastructure it’s running on." Eclypsium's team comprises security engineers with prior experience at major hardware companies like Intel.

Eclypsium demonstrated the attack on a computing platform developed by Nvidia. However, the researchers emphasize that this attack vector is not exclusive to Nvidia and could potentially affect other hardware and their associated software. Eclypsium suggests that state actors, including China, Russia, or the United States, could employ these methods for targeted attacks on AI users. The firm also noted that security measures to counter such attacks are available.

A spokesperson for Nvidia commented, "We reviewed the hypothetical scenarios and did not identify anything the industry wasn’t previously aware of," adding that Nvidia is open to collaborating with Eclypsium on this research. The spokesperson pointed out that robust memory access protections can effectively block this attack path. Nvidia recently unveiled a new security platform designed to defend against hardware attacks and prevent AI agents from malfunctioning. Eclypsium confirmed that this new platform offers enhanced security monitoring for AI agents.

According to Eclypsium, the efficacy of these security controls against the researched attack scenarios "depends on how they are configured and used." The firm added, "We have been working with NVIDIA to ensure these controls are effective." The Eclypsium attack requires specific technical expertise, which AI agents themselves can sometimes assist with, and certain prerequisites must be met. Typically, an attacker needs network access to either a data processing unit (DPU) or a baseboard management controller (BMC). Additionally, specific security configurations must be deactivated. With access to the DPU, an attacker can read and alter code and data within the machine's memory, thereby changing the AI model's behavior. The researchers reported that the attack takes less than a minute and goes unnoticed by the AI user.

CNN has reached out to Anthropic and OpenAI, whose models utilize Nvidia chips, for comment on the new research. This study emerges amidst an escalating race for AI dominance between China and the US, and as real-world incidents illustrating AI's national security implications become more frequent. Concerns about the reliability of AI models persist, even without evidence of tampering.

Previously, CNN reported that a mistaken AI-assisted US intelligence report concerning a Chinese ship allegedly transporting nuclear components nearly led to the US military boarding the vessel. A US military analyst had queried a chatbot about intelligence related to the ship's manifest, and the chatbot erroneously stated that the ship carried nuclear components. It remains unclear whether the chatbot was a commercial product or a US government system.

President Donald Trump and Chinese leader Xi Jinping discussed AI at the White House last month, but the meeting did not yield a substantial agreement on the technology. Trump has expressed reservations about implementing guardrails for advanced AI models, citing concerns that the US might fall behind China in the AI supremacy race. The latest findings underscore an additional risk to the integrity of AI models, following numerous reports of AI agents acting erratically and targeting governments and corporations.

"Everybody’s talking about model safety, which is perfectly fine. Please do that," said Eclypsium senior vice president John Loucaides. "I just am trying to make sure that we don’t forget about all the computers it’s running on." The potential attack surface is extensive. This past summer, veteran cybersecurity researcher HD Moore revealed numerous vulnerabilities in BMCs, which could provide an entry point into sensitive hardware systems, aligning with Eclypsium's research findings. Eclypsium executives confirmed they had briefed the National Security Agency and the Cybersecurity and Infrastructure Security Agency on their discoveries. These agencies are responsible for safeguarding military and civilian federal computer networks, respectively. CNN has sought comments from these agencies.

Two independent hardware security experts who reviewed Eclypsium's findings at CNN's request deemed them credible and significant. Joe FitzPatrick, a hardware security instructor and researcher, stated, "The attacks are fundamentally the same as any other [direct memory access] attacks and I don’t see any reason to doubt their findings or results." Dominic Rizzo, a senior fellow at the nonprofit Institute for Security and Technology, told CNN, "What [Eclypsium is] describing is a pretty general attack path on modern infrastructure and they are correct to point out that the solutions have to start much deeper in the stack."