LogParser-LLM: Advancing Efficient Log Parsing with Large Language Models

日志是无处不在的数字足迹，在系统诊断、安全分析和性能优化中发挥着不可或缺的作用。从日志中提取有用的信息很大程度上依赖于日志解析过程，该过程将原始日志转换为结构化格式以进行下游分析。然而，现代系统的复杂性和日志的动态性给现有的自动解析技术带来了重大挑战。大型语言模型（LLM）的出现开辟了新的视野。凭借其广泛的知识和上下文能力，LLM已经在各种应用中产生了变革性的影响。在此基础上，我们引入了LogParser-LLM，这是一个集成了LLM功能的新型日志解析器。这种结合无缝地融合了语义洞察力和统计细微差别，消除了超参数调整和标记训练数据的需要，同时通过在线解析确保了快速适应性。进一步深入探索，我们解决了解析粒度的复杂挑战，提出了一个新的度量标准，并整合了人类交互，使用户可以根据自己的特定需求进行粒度校准。我们的方法通过对Loghub-2k和大规模LogPub基准进行评估来证明其有效性。在对LogPub基准的评估中，每个数据集平均涉及360万条日志，在14个数据集上，我们的LogParser-LLM平均只需要272.5次LLM调用，达到了90.6%的分组准确度和81.1%的解析准确度。这些结果证明了该方法具有高效性和准确性，优于当前最先进的日志解析器，包括基于模式、基于神经网络和现有的LLM增强方法。

Log parsing is a complex task due to the dynamic nature of logs and the complexities of contemporary systems. This paper aims to introduce a novel log parser integrated with Large Language Models (LLMs) capabilities to address this challenge.

The key idea of the paper is to use LLMs to enhance log parsing, blending semantic insights with statistical nuances, and obviating the need for hyper-parameter tuning and labeled training data. The paper also proposes a new metric for parsing granularity and integrates human interactions to allow users to calibrate granularity to their specific needs.

The paper introduces LogParser-LLM, a novel log parser that achieves high efficiency and accuracy. It requires only 272.5 LLM invocations on average, achieving a 90.6% F1 score for grouping accuracy and an 81.1% for parsing accuracy. The paper evaluates the method on Loghub-2k and the large-scale LogPub benchmark. The method outperforms current state-of-the-art log parsers, including pattern-based, neural network-based, and existing LLM-enhanced approaches. The paper also addresses the challenge of parsing granularity and integrates human interactions to allow users to calibrate granularity to their specific needs.

Recent related work includes pattern-based, neural network-based, and existing LLM-enhanced log parsers. Some of the related papers include 'DeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning', 'LogMine: Fast Pattern Recognition for Log Analytics', and 'BERT-based log parsing method for large-scale log data analysis'.