Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety

Tomek Korbak ,

Mikita Balesni ,

Elizabeth Barnes ,

Yoshua Bengio ,

Joe Benton ,

Joseph Bloom ,

Mark Chen ,

Alan Cooney ,

Allan Dafoe ,

Anca Dragan ,

Scott Emmons ,

Owain Evans ,

David Farhi ,

Ryan Greenblatt ,

Dan Hendrycks ,

Marius Hobbhahn ,

Evan Hubinger ,

Geoffrey Irving ,

Erik Jenner ,

Daniel Kokotajlo ,

Victoria Krakovna ,

Shane Legg ,

David Lindner ,

David Luan ,

Aleksander Mądry ,

Julian Michael ,

Neel Nanda ,

Dave Orr ,

Jakub Pachocki ,

Ethan Perez ,

Mary Phuong ,

Fabien Roger ,

Joshua Saxe ,

Buck Shlegeris ,

Martín Soto ,

Eric Steinberger ,

Jasmine Wang ,

Wojciech Zaremba ,

Bowen Baker ,

Rohin Shah ,

Vlad Mikulik

AI Safety / AI Ethics AVAR RLAW

以人类语言进行“思考”的AI系统为AI安全提供了一个独特的机会：我们可以监控它们的推理过程（Chain of Thought，CoT）中是否有意图做出有害行为。与所有其他已知的AI监督方法一样，CoT监控并不完美，仍有一些不当行为可能未被察觉。尽管如此，这种方法展现出了一定的潜力，我们建议进一步研究CoT的可监控性，并在现有安全措施的基础上加大对CoT监控的投入。由于CoT的可监控性可能较为脆弱，我们建议前沿模型的开发者在开发过程中考虑各项决策对CoT可监控性的影响。

AI systems that "think" in human language offer a unique opportunity for AI safety: we can monitor their chains of thought (CoT) for the intent to misbehave. Like all other known AI oversight methods, CoT monitoring is imperfect and allows some misbehavior to go unnoticed. Nevertheless, it shows promise and we recommend further research into CoT monitorability and investment in CoT monitoring alongside existing safety methods. Because CoT monitorability may be fragile, we recommend that frontier model developers consider the impact of development decisions on CoT monitorability.