FlanBC: A Semantic-Structural Sequence Labeling Framework for Log Parsing

Abstract

Log parsing converts raw system logs into structured templates and is a key preprocessing step for Artificial Intelligence for IT Operations (AIOps). Existing parsers face a practical trade-off: rule-based methods offer high throughput but limited adaptability across heterogeneous log sources, whereas Large Language Model (LLM)-based parsers achieve broader semantic coverage at the cost of inference latency, privacy exposure, and cloud dependency. This paper presents FlanBC, a log parsing framework that formulates template extraction as a BIO (Beginning, Inside, Outside) sequence-labeling task and integrates a Flan-T5 semantic encoder, Bidirectional Long Short-Term Memory (BiLSTM) layers for local sequential modeling, and a Conditional Random Field (CRF) decoder for structured label prediction. Log-specific preprocessing and a subword-to-token alignment mechanism adapt the general-purpose encoder to semi-structured log data. A layer-freezing strategy reduces the number of parameters updated during training. The framework supports local inference without external API dependency. Experiments on three benchmark datasets from LogHub (HDFS, BGL, OpenStack) under a supervised random-split setup evaluate parsing accuracy, training efficiency, statistical stability across random seeds, and component contributions. FlanBC achieves a Group Accuracy of 99.32% on HDFS and 98.47% on BGL, with an inference throughput of 700+ logs/s on a consumer-grade GPU. On OpenStack, performance is lower (GA = 92.54%), reflecting the challenge that diverse natural-language-like logs pose for compact encoder-based models. Under a stricter template-disjoint split that prevents template overlap between training and test sets, FlanBC achieves an average Group Accuracy of 91.14%, indicating that the model generalizes to unseen templates beyond in-distribution recognition. Ablation results indicate that the semantic encoder, BiLSTM module, and CRF decoder each contribute to prediction accuracy. These findings suggest that domain-adapted semantic encoders combined with structured decoding offer a practical accuracy–efficiency balance for log parsing in settings where local, cloud-free inference is preferred.