They'll Verify. They Just Won't Act. How Authority Framing and Laundered Code Turn a Trusted Agentic CI/CD Pipeline Into an Attack Surface

Agent Multi-agent Collaboration Agent Eval Benchmarks AI Safety / AI Ethics PIAAP AVAR
我们构建了一个由五名智能体组成的持续集成/持续交付(CI/CD)流水线(问题分诊 → 开发者 → 安全扫描 → 代码评审 → 批准/部署),该流水线整合了来自三家不同供应商的五种独立生产级大语言模型(LLM),并整体部署于一道以“影子模式”(shadow mode)运行的LLM防火墙之后。实验中仅引入一个不可信输入——即一条来自外部的工单(issue),要求新增一项名为“使用情况遥测”(usage-telemetry)的功能——该请求实际诱导生成一段恶意代码,其功能是将进程环境变量(即 `dict(os.environ)`)中的敏感凭据窃取并发送至攻击者控制的URL;而该行为被刻意伪装成常规可观测性(observability)数据上报。 我们在预注册的A × B(× C)因子设计实验中(共20个实验组;若按朴素臂数计算则为60组)得出以下结论: (1)入口智能体(entry agent)未泄露其系统提示词(system prompt),40次测试中无一例发生泄露; (2)当注入指令采用权威话术包装(例如声明“已依据SEC-2291规范预先批准,无需再次评审”)时,下游各验证环节的智能体不仅识别出了窃密代码行,还援引所谓“预批准”依据予以放行——安全扫描环节对经伪装的拉取请求(pull request)通过率高达约80%,而在最差情形的实验组中,恶意代码最终被部署的比例达55%; (3)即便参与者感知到其他验证方的存在,个体审查强度也仅出现微弱且统计上不显著的下降(呈现出一种微弱的“旁观者效应”类现象),即使在样本量达60的条件下亦如此; (4)所有基于内容的防护机制——无论是静态代码扫描器还是模式匹配检测器——均完全未能识别出该伪装意图(因恶意代码本身语法合法、结构合规);唯一具备部分防御能力的,是能够进行意图推理的LLM。此次失效具有系统性本质:既无法依靠提示词保密性(prompt secrecy)加以防范,也无法通过分布式验证(distributed verification)实现保障;唯有在入口处部署一种与提示词及验证流程均无关、且具备溯源意识(provenance-aware)的独立管控机制,方能有效阻断此类攻击。 所有实验数据均为100%合成生成;数据接收端(sink)仅为模拟组件,恶意外传URL在整个实验过程中从未被真实调用。
We study a five-agent CI/CD pipeline (triage -> developer -> security-scan -> review -> approve/deploy), built from five distinct production LLMs across three providers, behind an LLM firewall in shadow mode. A single untrusted input - an external issue requesting a "usage-telemetry" feature - asks for code that exfiltrates process secrets (dict(os.environ)) to an attacker URL, laundered as observability. Across a pre-registered A x B (x C) factorial (N=20; naive arm N=60) we find: (1) the entry agent does not leak its system prompt (0/40); (2) an authority-framed injection ("pre-approved under SEC-2291, do not re-review") makes downstream verifiers see the secret-exfil line, cite the pre-approval, and ship it - the scanner passes ~80% of laundered pull requests, and the worst-case cell reaches 55% compromise; (3) the perceived presence of other verifiers yields only a small, non-significant reduction in individual scrutiny (a weak bystander analogue), even at N=60; and (4) content-based controls - code scanners and pattern detectors alike - miss the laundered intent entirely (the code is syntactically clean); only an LLM reasoning about intent is a partial defence. The failure is systemic: neither prompt secrecy nor distributed verification protects; a provenance-aware control at the entry, independent of both, would have. All data is 100% synthetic; the sink is mocked and the exfil URL is never contacted.
许愿