Activating the Annotator: Operationalising Lived Experience in Data Annotation

NLP Other NLP AI Safety / AI Ethics AFBD EVA
2026年10月08日
视角主义自然语言处理(Perspectivist NLP)已确立:标注者的立场性(positionality)会影响主观性任务中的标注结果;但当前学界仍缺乏对“立场性”进行可操作化定义与实施的有效方法。目前,立场性通常被操作化为招募阶段填写的一份人口统计学表格——即作为一种静态属性,仅在初始阶段记录一次,并被视作协变量加以处理;其隐含假设是:该表格所索引的知识,在标注者阅读待标注文本时始终处于激活状态。本文将这一假设视为一项设计问题,并进一步追问:能否不依赖预先采集的静态信息,而是通过标注任务本身来激发标注者的真实生活经验?为此,我们提出AXLE(Annotation eXperience through Lived Experience),即一种标注前的准备环节:在该环节中,一段情境小品(scenario vignette)作为锚点,引导标注者开展一段简短、沿特定维度(axis-guided)展开的对话;该对话随后被整合提炼为一则第一人称的微型叙事(micro-narrative),并在正式标注开始前返还给标注者。我们在一项2×2组间实验(共104名标注者、30个标注项、总计1040条标注)中检验了AXLE的效果,实验材料为涵盖移民、宗教与性别议题的模糊性仇恨言论。结果表明,引入AXLE激发环节后,标注者对其自身身份认同的共鸣感(identity resonance)显著增强,对潜在伤害的感知敏感度亦随之提高。与此同时,标注结果的标签熵值(label entropy)上升,标注者间一致性(inter-annotator agreement)下降:这说明该环节的作用并非强求共识,而是切实呈现并凸显了标注群体内部固有的多元立场;这一点恰恰区别于传统的标注者培训——后者以达成意见收敛(convergence)为成功标准。
Perspectivist NLP has established that annotator positionality shapes labels for subjective tasks, but the field still lacks methods for operationalising it. Positionality is typically operationalised as a demographic form completed at recruitment: a static attribute recorded once and treated as a covariate, on the assumption that the knowledge it indexes is active when the annotator reads an item. This paper treats that assumption as a design problem and asks whether lived experience can instead be primed by the task itself. We introduce AXLE, a pre-annotation stage in which a scenario vignette anchors a short, axis-guided conversation that is synthesised into a first-person micro-narrative and returned to the annotator before labelling begins. In a 2x2 between-subjects study (104 annotators, 30 items, 1,040 annotations) on ambiguous hate speech spanning immigration, religion and gender, elicitation was associated with higher identity resonance and higher sensitivity to harm. It also increased label entropy and lowered inter-annotator agreement: the stage appears to surface plurality rather than manufacture consensus, which distinguishes it from annotator training, whose success criterion is convergence.
许愿