The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspective

近年来，大型语言模型（LLMs）得到了快速发展。基于强大的LLMs，多模态LLMs（MLLMs）将模态从文本扩展到更广泛的领域，因其更广泛的应用场景而受到广泛关注。由于LLMs和MLLMs依赖于大量的模型参数和数据来实现新兴能力，因此数据的重要性正在得到越来越广泛的关注和认可。通过追踪和分析最近针对MLLMs的数据导向工作，我们发现模型和数据的发展不是两条独立的道路，而是相互关联的。一方面，更广泛和高质量的数据有助于提高MLLMs的性能，另一方面，MLLMs可以促进数据的发展。多模态数据和MLLMs的共同发展需要明确的视角，即1）在MLLMs的哪个开发阶段可以采用特定的数据导向方法来增强哪些能力，以及2）通过利用哪些能力并扮演哪些角色，模型可以为多模态数据做出贡献。为了促进MLLM社区的数据-模型共同发展，我们从数据-模型共同发展的角度系统地审查了与MLLMs相关的现有工作。与此调查相关的定期维护的项目可在https://github.com/modelscope/data-juicer/blob/main/docs/awesome_llm_data.md上访问。

MLLMs rely on vast amounts of model parameters and data to achieve emergent capabilities, thus the co-development of multi-modal data and MLLMs requires a clear view of how models can contribute to multi-modal data and how data can enhance MLLMs' performance.

The development of models and data for MLLMs is interconnected, and vaster and higher-quality data can contribute to better performance of MLLMs, while MLLMs can facilitate the development of data. The paper systematically reviews existing works related to MLLMs from the data-model co-development perspective to promote the co-development of multi-modal data and MLLMs.

The paper traces and analyzes recent data-oriented works for MLLMs and provides a regularly maintained project associated with the survey. The experiment design, datasets used, and open-source codes are not mentioned in the abstract. The paper emphasizes the importance of co-development of multi-modal data and MLLMs and provides a clear view of how to enhance MLLMs' performance and develop multi-modal data.

The paper does not list any related works in the abstract.