Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasters

Xiao Ye ,

Jacob Dineen ,

Evan Zhu ,

Shijie Lu ,

Kevin Song ,

Ben Zhou

LLM HDCE Other LLM NLP RAG DCRS Benchmarks

2026年07月15日

预测者通常通过回溯测试（backtesting）进行评估，即重新播放已知结果的问题，并评定系统在结果揭晓前本应给出的概率。然而，对于大语言模型（LLM）而言，此类测试存在两条信息泄露路径：其一，具备检索能力的模型可能调取出事件发生后撰写的报道，从而将预测任务退化为简单查表；其二，每一代新模型的训练数据都更接近事件发生时间，因此对去年的模型而言尚属未来的某个问题，却已悄然落入今年模型的训练数据之中。无论哪种情况，该测试实际评估的是模型的记忆能力（recall），却声称衡量的是其前瞻性判断能力（foresight）。为此，我们提出“后见式评估法”（Hindcast），通过将模型置于某一选定的过去时刻 $t_0$（即结果尚未产生、且尚未通过任一渠道进入公共信息流的时间点）来严格封堵上述两类信息泄露。具体而言，Hindcast 以 Polymarket 预测市场中已完结的交易为基准，回放其历史行情，并仅向模型提供一份固定截取的 Reddit 公共帖子快照——其中所有帖子均须早于 $t_0$ 发布；随后，模型所作的每一项预测，均需同时对照两个标准进行评分：一是该事件最终的实际结果，二是 $t_0$ 时刻 Polymarket 市场自身的价格——这一价格本身即是由人类基于与模型完全相同的过往信息所作出的预测。由于每个预测市场的截止时间 $t_0$ 是单独设定的，且所用 Reddit 快照永久冻结不变，因此当模型能力提升后，该评估框架可直接复用于新增市场，而不会因数据过时而失效。一旦彻底阻断信息泄露，检索能力对多数模型仍有助益，但仅限于 Reddit 在事件发生前已有相关讨论的情形；而在档案中仅有猜测性内容的情况下，检索反而会损害预测表现。

Forecasters are evaluated by backtesting, which replays resolved questions and grades the probability the system would have assigned before the outcome was known. For LLMs, two channels leak the answer into this test. A model that retrieves can surface reports written after the event, turning forecasting into a lookup, and each new model is trained on data closer to the event, so a question that lay in the future for last year's models sits inside this year's training data. Either way, the test grades recall while claiming to grade foresight. We introduce Hindcast, which closes both leaks by grading a model as if it stood at a chosen past date $t_0$, before the outcome existed in either channel. Hindcast replays resolved Polymarket prediction markets against a frozen snapshot of public Reddit, lets the model read only posts written before $t_0$, and scores each forecast against both what happened and the market's own price at $t_0$, itself a human forecast made from the same past information. Because the cutoff is set per market and the snapshot never changes, the evaluation re-runs on new markets as models improve, without going stale. Once the leak is closed, retrieval still helps most models, but only where Reddit discussed the event beforehand. Where the archive carried only speculation, retrieval hurts.

许愿