With the rapid development of computer and network technology, network and software logs generated by a multitude of devices contain a wealth of knowledge and serve as a critical resource for intelligent fault diagnosis and efficient system operations. In recent years, various deep learning methods and the pre-training and fine-tuning paradigm of large language models have achieved significant success in log understanding. However, most existing approaches directly adapt models designed for natural language, overlooking the unique characteristics inherent to log data, such as its distinct vocabulary distribution, structural patterns, and semantic expressions. Concurrently, some template-based methods lack flexibility and are limited in their ability to mine deep semantic information. Therefore, we propose LogPISA, an improved pre-training and tuning pipeline for log understanding with invariant and semantic-aware objectives. Our framework employs a hybrid attention mechanism, combining standard self-attention with our novel Keyword-Aware Sparse Attention to enable the model to capture critical signals more efficiently. During the pre-training phase, we introduce two innovative self-supervised tasks: a non-contrastive learning task based on permutation invariance to capture the flexible ordering within log blocks, and a contrastive learning task based on log summarization to guide the model to focus on core semantics over superficial textual forms. Experimental results on several public benchmarks demonstrate that our model achieves excellent performance on downstream anomaly detection tasks. This validates that our proposed framework learns more robust and generalizable log representations, providing a high-quality representational foundation for various downstream log analysis tasks.
Lanlan Rui, Yuanrui Yang, Peng Yu et al.· IEEE Transactions on Network...· 0 citations
In agentic AI-enabled edge computing, decentralized federated learning (DFL) leverages peer-to-peer model aggregation to improve the performance of on-device large language models (LLMs) without introducing a single point of failure, thereby enhancing local agents’ capabilities for decision-making. To protect the right to be forgotten for each agent, as required by data regulations such as the General Data Protection Regulation (GDPR), federated unlearning aims to remove the influence of a target agent’s data from the trained LLM while preserving model utility. However, existing federated unlearning methods predominantly assume centralized architectures and face two critical challenges when extended to decentralized federated learning systems: 1) requiring all remaining agents to participate in the unlearning process incurs prohibitive overhead; and 2) updating all Low-Rank Adaptation (LoRA) modules indiscriminately leads to excessive resource consumption while potentially degrading model performance. To tackle these challenges, we propose a dual-level selective unlearning framework (DSU) for decentralized federated learning. At the agent level, DSU selects retained agents by matching historical LoRA update sketches, label sketches, and exposure to the withdrawn agent. Meanwhile, we derive a sensitivity score from the DFL training trajectory that measures each LoRA module’s accumulated influence from the target agent via the energy of its effective weight changes, and then update only the most sensitive modules while freezing the rest. Theoretical analysis relates selective unlearning to full-participation unlearning, and experiments show that DSU reduces the audited target-agent influence while preserving retained utility and reducing both participating agents and updated LoRA modules in the evaluated settings. The public reproducibility package is available at https://github.com/DGL-codes/LLM-DFL
Zhiqiang Xie, Yijing Lin, Zhipeng Gao et al.· IEEE Transactions on Cogniti...· 0 citations