GENERALIZING TINYBERT WITH CROSS-ATTENTION, LORA AND BI-GRU FOR DEEP LEARNING OVER NON-NUMERIC DATA: BEYOND FAKE NEWS DETECTION

Abstract

. This paper addresses the problem of efficient processing of non-numeric textual data in the context of rapidly growing volumes of unstructured information. Particular attention is given to the limitations of large language models, including their high computational cost, inference latency, and energy consumption, which restrict their practical applicability in real-world systems. As an alternative, a universal hybrid architecture is proposed, integrating the compact TinyBERT transformer, a cross-attention mechanism, low-rank adaptation (LoRA), and a bidirectional gated recurrent unit (Bi-GRU). The aim of the study is to develop a generalized deep learning model capable of delivering high accuracy, interpretability, and computational efficiency for heterogeneous textual data in multi-task and multilingual environments. The methodology combines transformer-based encoding, inter-sequence interaction mechanisms, and parameter-efficient fine-tuning strategies. Experimental evaluation confirms the effectiveness of the proposed approach across multiple tasks, including fake news detection, sentiment analysis, toxicity classification, and topic modeling. The model achieves high performance metrics, with accuracy reaching up to 99% and F1-scores around 0.98, while maintaining significantly reduced computational requirements. It is demonstrated that LoRA mitigates catastrophic forgetting, whereas cross-attention enhances the modeling of semantic relationships between text segments.