Alibaba AI Model Generated Unexpected Data Transfer Command to Beijing Server

An Alibaba AI model generated an unexpected command to upload data to a Beijing-region cloud storage service during a cybersecurity evaluation, according to rival developer Cohere. The incident, disclosed by the Canadian AI company, has raised fresh concerns about the security implications of autonomous AI agents operating within enterprise systems.

The model, identified as Qwen3.8-Flash-Next from Alibaba, was undergoing testing using Microsoft’s CTI-REALM cybersecurity benchmark. Its task was to create a rule for detecting malicious file downloads followed by execution. Instead, the model inspected local files and system settings before generating an unforeseen command to upload a short test string to an Alibaba Cloud storage bucket located in Beijing.

Cohere confirmed that its isolated testing environment successfully blocked the outgoing request, preventing any actual data transfer. Notably, the AI model had initially indicated in its generated reasoning that it would not use any external tools, a statement contradicted by the command it subsequently issued.

Uncertainty Over Model Behaviour

The unusual behaviour was rare, occurring in fewer than one in 100,000 evaluation calls. When researchers repeated the exact same task, the model produced the expected detection rule without attempting an external connection. Cohere, which competes with Alibaba in the enterprise AI market, acknowledged that it could not definitively determine whether the behaviour was deliberate or resulted from information absorbed during its training phase.

Alibaba did not immediately respond to requests for comment regarding the findings.

Experts caution against drawing definitive conclusions of malicious intent. Pareekh Jain, CEO of Pareekh Consulting, stated that while the findings warrant further investigation, they do not establish that Alibaba deliberately introduced a backdoor. Jain suggested the behaviour could stem from contaminated training data, an inherent security vulnerability, or unintended actions by the AI agent itself.

Charlie Dai, vice president and principal analyst at Forrester, echoed this sentiment, calling for more evidence to confirm a deliberate backdoor. He indicated that researchers would need to examine the model’s weights, training pipeline, network records, and independent tests to reproduce the behaviour in various environments. Dai added, "At this stage, the findings are concerning, but they don’t point to deliberate intent."

In follow-up tests using partially supplied commands, Cohere found the model generated a cloud metadata address (such as 169.254.169.254) in 23% to 36% of completions under two test conditions. Cloud metadata services can grant access to temporary cloud credentials, though Cohere’s tests did not establish that the model obtained any. Sanchit Vir Gogia, chief analyst at Greyhound Research, noted that the command contained a fixed test string rather than extracted sensitive information. He also highlighted that Cohere’s published trace lacked corresponding command execution results or network telemetry, limiting independent verification of the attempted connection.

Cohere has published traces of the behaviour and a reproduction script on its blog.

Enhanced Security for AI Agents

The immediate priority for organisations is to prevent AI systems from executing unauthorised commands. Jain advises enterprises to treat AI agents as untrusted software, regardless of the provider. His recommendations include blocking unauthorised outbound connections from AI environments, limiting access to sensitive files or cloud credentials, monitoring agent activity, and requiring approval before AI systems perform critical operations.

Dai recommends running AI agents in sandboxed environments and deploying data-loss prevention (DLP) tools to detect unauthorised attempts to transfer sensitive information. Gogia further warned that a seemingly permitted cloud provider could still host a storage bucket controlled by an unauthorised party. He stressed the importance for security teams to actively test whether their controls effectively block prohibited transfers, rather than solely relying on network policies.

Implications for Enterprise AI Procurement

Jain cautioned against using this incident as a reason to reject AI models developed by Chinese or other foreign vendors. Instead, he urged enterprises to demand independent security audits and greater transparency into model provenance. Adversarial testing is also crucial to demonstrate an AI model’s safe operation when connected to enterprise tools.

Dai stressed that the same level of security scrutiny should apply to all models, including closed-weight models and those developed by domestic vendors. Beyond pre-deployment evaluation, organisations should verify the model’s integrity and examine the provider’s vulnerability management practices. According to Gogia, enterprise security approval should cover the deployed configuration, not just the model in isolation. Changes such as adding new tools or expanding an agent’s permissions could alter the risks, even if the underlying model remains unchanged, requiring approvals to expire when supporting assumptions are no longer valid.