A Review: Context Dependent and Context Independent Word Embedding Techniques

Abstract

Word embedding techniques have become a fundamental component of Natural Language Processing (NLP) by enabling machines to learn meaningful semantic and syntactic representations of text. Over the past decade, embedding methods have evolved from static representations, such as Word2Vec, GloVe, and FastText, to contextual models including ELMo, BERT, GPT, and their transformer-based variants. This paper presents a systematic review of contextindependent and contextual word embedding techniques, emphasizing their architectural evolution, contextual understanding capabilities, computational complexity, and performance across diverse NLP applications. A structured literature review methodology was adopted to analyse recent studies and identify key developments in embedding techniques. The review critically compares existing approaches with respect to polysemy handling, semantic representation, computational efficiency, and downstream task performance. Furthermore, recent advancements in sentence embeddings and transformer-based embedding models are discussed to provide a comprehensive overview of the current research landscape. Finally, the paper highlights existing research gaps and outlines promising future research directions for developing efficient, scalable, and context-aware embedding models. The findings provide practical guidance for researchers in selecting appropriate embedding techniques for modern NLP applications.