LogPISA: An Improved Pre-Training and Tuning Pipeline for Log Understanding With Invariant and Semantic-Aware Objectives

Abstract

With the rapid development of computer and network technology, network and software logs generated by a multitude of devices contain a wealth of knowledge and serve as a critical resource for intelligent fault diagnosis and efficient system operations. In recent years, various deep learning methods and the pre-training and fine-tuning paradigm of large language models have achieved significant success in log understanding. However, most existing approaches directly adapt models designed for natural language, overlooking the unique characteristics inherent to log data, such as its distinct vocabulary distribution, structural patterns, and semantic expressions. Concurrently, some template-based methods lack flexibility and are limited in their ability to mine deep semantic information. Therefore, we propose LogPISA, an improved pre-training and tuning pipeline for log understanding with invariant and semantic-aware objectives. Our framework employs a hybrid attention mechanism, combining standard self-attention with our novel Keyword-Aware Sparse Attention to enable the model to capture critical signals more efficiently. During the pre-training phase, we introduce two innovative self-supervised tasks: a non-contrastive learning task based on permutation invariance to capture the flexible ordering within log blocks, and a contrastive learning task based on log summarization to guide the model to focus on core semantics over superficial textual forms. Experimental results on several public benchmarks demonstrate that our model achieves excellent performance on downstream anomaly detection tasks. This validates that our proposed framework learns more robust and generalizable log representations, providing a high-quality representational foundation for various downstream log analysis tasks.