What is the Role of Small Models in the LLM Era: A Survey

大型语言模型（LLMs）在推进人工通用智能（AGI）方面取得了显著进展，导致了越来越大的模型的发展，如GPT-4和LLaMA-405B。然而，增加模型大小会导致指数级的计算成本和能源消耗，使得这些模型对于资源有限的学术研究人员和企业来说不切实际。同时，小型模型（SMs）经常在实际环境中使用，尽管它们的重要性目前被低估了。这引发了关于小型模型在LLMs时代的作用的重要问题，这个话题在以前的研究中受到了有限的关注。在这项工作中，我们从协作和竞争两个关键角度系统地研究了LLMs和SMs之间的关系。我们希望这项调查为从业者提供有价值的见解，促进对小型模型的贡献的更深入理解，并促进更有效地利用计算资源。代码可在https://github.com/tigerchen52/role_of_small_models上获得。

LLMs have high computational costs and energy consumption, making them impractical for academic researchers and businesses with limited resources. This paper aims to examine the role of Small Models (SMs) in the era of LLMs.

The paper systematically examines the relationship between LLMs and SMs from the perspectives of collaboration and competition. It suggests that SMs have significant contributions and should be used more efficiently to save computational resources.

The paper provides insights for practitioners on the contribution of SMs and promotes more efficient use of computational resources. It suggests that SMs can be used in collaboration with LLMs to improve performance and reduce computational costs. The experiments are designed to compare the performance and efficiency of LLMs and SMs on various tasks. The code is available on Github.

Recent related studies include 'The Power of Scale for Parameter-Efficient Prompt Tuning' and 'Small and Practical Language Models are Possible'.