Guardrails or Roadblocks? Effects of Pedagogical Style and Context Awareness in AI Teaching Assistants for Programming

Agent Human Agent Interaction AI Apps / Applied AI LPM
由大语言模型(LLM)驱动、并配备教学约束机制(pedagogical guardrails)的人工智能教学助手(AI TA),正日益被整合进编程课程中,为学生提供可扩展的即时提示、概念性讲解以及面向代码层面的反馈。然而,这些教学约束机制本身也可能带来使用阻力:倘若学生感到所获支持过于严苛,或与自身当前的学习进展缺乏适配性与情境相关性,他们便可能绕过经批准的教学工具,转而求助于通用型大语言模型。为探究AI TA的设计特征如何影响学生的学习体验,我们在一门入门级编程课程中开展了一项随机对照实验,共招募132名学生参与。所有学生需完成三项与代码编写及程序调试相关的任务,并被随机分配至四种AI TA条件之一;这四种条件在两个维度上形成正交变化:教学引导风格(苏格拉底式引导 vs. 直接讲授式)和情境感知能力(无上下文信息 vs. 全面掌握题目要求及学生当前解题方案)。我们系统考察了学生对AI TA的主观评价、实际交互行为,以及任务后测中所体现出的理解程度证据。结果显示,学生对“具备完整情境感知能力的苏格拉底式AI TA”的评价最低,认为其在支持任务完成方面所提供的帮助显著不足。从描述性统计来看,该组学生在交互过程中表现出最高的压力水平、最高的外部大语言模型使用率,以及最低比例的、能体现完全理解的任务后解释——尽管上述差异均未达到统计学显著性水平。本研究发现表明:配备教学约束机制的AI教学助手并非天然更有利于学习;其实际教学效能,取决于教学引导策略与情境感知能力之间能否实现恰当平衡——而这种平衡必须让学生真切感受到其有用性、支持性,并愿意持续使用。
AI teaching assistants (AI TAs) backed by large language models (LLMs) and pedagogical guardrails are increasingly being integrated into programming courses, providing students with scalable access to hints, conceptual explanations, and code-level feedback. However, guardrails may also create friction. If students feel that the support provided is overly restrictive or poorly contextualized to their current progress, they may bypass approved tools for general-purpose LLMs. To investigate how AI TA design affects students' learning experiences, we conducted a randomized controlled trial with 132 students in an introductory programming course. Students completed three tasks related to code-writing and debugging and were randomly assigned to one of four AI TAs varied across two dimensions: pedagogical guidance style (Socratic vs. Direct instruction) and context awareness (no context vs. full context of the problem and student solution). We examined students' perceptions, interaction behaviors, and evidence of post-task comprehension. Students rated the Socratic AI TA with full context least favorably, reporting significantly lower perceived support for task completion. Descriptively, this condition also showed the highest observed interaction stress, the highest rate of external LLM use, and the lowest proportion of post-task explanations demonstrating full comprehension, though these differences were not statistically significant. These findings suggest that guardrailed AI TAs are not automatically better for learning. Instead, their effectiveness depends on how pedagogical guidance and contextual awareness are balanced in ways that students experience as useful, supportive, and worth continuing to use.
许愿