DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

AK's Picks LLM RLHF Latent Reasoning CoT ML RL
3182
2025年01月22日
我们介绍了第一代推理模型DeepSeek-R1-Zero和DeepSeek-R1。DeepSeek-R1-Zero是一个通过大规模强化学习（RL）训练的模型，在初步步骤中没有使用监督微调（SFT），它展示了出色的推理能力。通过强化学习，DeepSeek-R1-Zero自然地展现出许多强大且有趣的推理行为。然而，它也遇到了一些挑战，如可读性差和语言混用问题。为了解决这些问题并进一步提升推理性能，我们引入了DeepSeek-R1，该模型在强化学习之前结合了多阶段训练和冷启动数据。DeepSeek-R1在推理任务上的表现与OpenAI-o1-1217相当。为了支持研究社区，我们开源了DeepSeek-R1-Zero、DeepSeek-R1以及从DeepSeek-R1基于Qwen和Llama蒸馏出的六个密集模型（分别为15亿、70亿、80亿、140亿、320亿和700亿参数）。
We introduce our first-generation reasoning models, DeepSeek-R1-Zero and DeepSeek-R1. DeepSeek-R1-Zero, a model trained via large-scale reinforcement learning (RL) without supervised fine-tuning (SFT) as a preliminary step, demonstrates remarkable reasoning capabilities. Through RL, DeepSeek-R1-Zero naturally emerges with numerous powerful and intriguing reasoning behaviors. However, it encounters challenges such as poor readability, and language mixing. To address these issues and further enhance reasoning performance, we introduce DeepSeek-R1, which incorporates multi-stage training and cold-start data before RL. DeepSeek-R1 achieves performance comparable to OpenAI-o1-1217 on reasoning tasks. To support the research community, we open-source DeepSeek-R1-Zero, DeepSeek-R1, and six dense models (1.5B, 7B, 8B, 14B, 32B, 70B) distilled from DeepSeek-R1 based on Qwen and Llama.
许愿