CodeLite: A Low-Cost Framework for Code Classification Tasks via Enhanced Code Representations and Lexical Fusion

Abstract

The prevailing approach in software engineering for code classification tasks is to rely on large-scale code language models such as StarCoder and CodeLlama. While these models have demonstrated strong performance, they come with the price of intensive computations that raise energy concerns and limit their deployment in resource-constrained environments. In this work, we propose ‘CodeLite’, a relatively lightweight framework that integrates moderately sized code models, such as CodeBERT, with traditional frequency-based techniques, achieving better performance and computational efficiency compared to larger models. Our approach comprises multiple stages, starting with the enhancement of encoder-based code models, where we leverage their intermediate-layer information to capture richer lexical and syntactic signals beyond the default [CLS] token. In parallel, we employ a regex-enhanced TF-IDF component tailored for source code to capture complementary frequency-based patterns. These representations are then combined with the learned code embeddings through suitable fusion techniques for final decision-making. Evaluated across multiple downstream tasks, including programming language identification, authorship attribution, and AI plagiarism, CodeLite consistently outperforms strong baseline models, achieving up to 4.2% absolute accuracy improvement over other approaches and 11.8% incremental gain over ablation components while incurring significantly lower training costs. We further conduct several statistical tests and ablation studies to validate the contribution of each component in the proposed framework. Overall, our results demonstrate that a careful combination of lightweight models along with task-aware feature enhancements can serve as a practical and efficient alternative to heavyweight language models for various software engineering tasks.