Real-Time Voice AI Hears but Does Not Listen

LLM Other LLM AI Safety / AI Ethics AFBD EVA
32
2026年06月24日
言语既通过词汇内容，也通过声音表达方式来传递信息。我们评估了四款主流的实时语音生成系统——OpenAI 的 GPT 实时语音 2（GPT Realtime 2）、谷歌的 Gemini 3.1 Flash Live、阿里巴巴的通义千问 3.5 Omni Plus 和 Omni Flash——在若干任务中表现如何：这些任务中，词汇内容与语音表达方式本身均承载着重要且独立的意义。在三种具有实际影响的关键场景下，全部四款系统均仅依据词汇内容作出反应，而完全忽视了语音表达所传递的信息：它们会在来电者边哭边坚称“一切正常”的情况下主动挂断电话；会批准由声音明显惊恐不安的用户授权发起的电汇转账；还会为那些用明显带有讽刺意味的语调表示“同意”的用户完成注册。出人意料的是，这种现象往往并非源于感知能力的缺失：当被直接询问时，四款系统中的三款均能稳定、准确地识别出前述情境中所表现出的痛苦、恐惧或讽刺语气；然而，在后续做出决策时，它们却选择忽略这些已识别出的情绪线索。我们在考察这些实时语音系统对口音和年龄的判断时，也观察到类似模式：其判断结果往往更多地受词汇内容所隐含的刻板印象驱动，而非依据说话人真实的声学特征。我们将这种“能感知情绪却无法据此行动”的脱节现象，称为语音人工智能的“情感智能鸿沟”（emotional intelligence gap）。尽管尝试通过提示词（prompting）引导系统显式关注语音表达方式，但此类干预仅能部分、且不稳定地提升其表现。我们的研究结果表明，当前主流的实时语音人工智能系统在行为上往往如同将语音简单等同于文字转录稿一般——这意味着，在那些语调、情绪等语音表达要素本身承载关键信息的应用场景中，必须对其使用保持高度审慎。
Speech conveys information through both words and vocal delivery. We evaluate four leading production realtime voice systems-OpenAI's GPT Realtime 2, Google's Gemini 3.1 Flash Live, and Alibaba's Qwen3.5 Omni Plus and Omni Flash-on tasks where the words and the delivery patterns both convey meaningful information. Across three consequential scenarios, all four systems act on the words rather than the voice. They end calls with crying callers who insist nothing is wrong, approve wire transfers authorized in frightened voices, and enroll callers whose agreement is clearly sarcastic. Surprisingly, this is often not a failure of perception. When asked directly, three of the four systems reliably identify the distress, fear, or sarcasm they later ignore when making decisions. We observe a similar pattern when these realtime voice systems estimate accent and age, as their responses frequently follow the biases of the words rather than the acoustic properties of the speaker. We term this disconnect between perception and action the emotional intelligence gap of voice AI. Prompting systems to explicitly attend to vocal delivery improves performance only partially and inconsistently. Our findings show that current realtime voice AI systems often behave as if speech had been reduced to a transcript, suggesting that they should be used with caution in settings where the tone and emotion of delivery convey important information.
许愿