Research on Fault Prediction Method of Enterprise Data Pipeline Based on ETL Process Logs

Abstract

As enterprise digital systems continue to operate, the task dependencies and execution states in large-scale data processing workflows become increasingly complex. The rich operational information contained in ETL process logs provides crucial information for data pipeline fault prediction. However, traditional methods typically rely on single log features or local state information, making it difficult to fully characterize the relationships between tasks, event evolution processes, and potential fault modes. This paper proposes an enterprise data pipeline fault prediction method based on ETL process logs. By unifying the representation of heterogeneous log events, a feature modeling mechanism oriented towards process instances is constructed. In the feature learning stage, the method integrates numerical attributes, category information, task states, component features, and time information to form event representations with contextual semantics. Simultaneously, a dependency-enhanced event graph is introduced to describe the execution relationships between tasks, and a relationship-time-aware graph attention mechanism is combined to capture the interactive influence between different event nodes, thereby enhancing the model's ability to understand complex operational structures. To address the issue that fault behavior exhibits different forms at different time scales, a multi-scale time deviation modeling mechanism is further constructed. By analyzing local changes, mid-term fluctuations, and long-term evolution characteristics, key dynamic information related to risk states is extracted. Building upon this foundation, a scale-adaptive fusion strategy is employed to integrate multi-source features, and evidence-selective instance aggregation is used to obtain a risk representation oriented towards the entire process, thereby enabling enterprise data pipeline fault state prediction. Experimental results demonstrate that the proposed method exhibits good overall performance in enterprise data pipeline fault prediction tasks, effectively utilizing structural and temporal information from process logs to enhance fault risk identification capabilities.