Analysis of the pre-trained embedding geometry of a small sentence transformer (SBERT) and classic SLM reveals that pre-trained embedding geometry is associated with classification performance and reveals a counterintuitive finding that a structured input that would help a human reader does not improve the SLM performance.
Abstract
Categorising invoices into the correct General Ledger (GL) code underpins financial reporting and tax compliance. This is a skilled accounting judgement rather than a routine task: the correct category depends subtly on the nature of the purchasing business, the vendor and the invoice text. Whilst AI is increasingly being adopted across industries to automate tasks, including invoice categorisation, implementations built on in-house small language models (SLMs) can simultaneously reduce cost and improve data security, confidentiality, and interpretability. We investigate this approach by first analysing the pre-trained embedding geometry of a small sentence transformer (SBERT) and classic SLM (DeBERTa). The sentence-embedding space of this financial corpus is globally anisotropic but composed of locally isotropic clusters, extending prior token-level findings to sentence embeddings in a financial setting, and these clusters are strongly correlated with the vendor identity. SBERT fine-tuned on a single GPU reaches 0.96 accuracy on invoice classification, above both a zero-shot LLM and a vendor identity baseline, increasing performance for smaller, challenging categories and new clients. For this important generalisation problem, SBERT reaches 0.9 F1 with roughly 100 client-specific invoices, showing that an in-house SLM implementation is promising. Combining these results with geometric analysis shows that pre-trained embedding geometry is associated with classification performance and reveals a counterintuitive finding that a structured input that would help a human reader does not improve the SLM performance.
A hybrid detection framework which combines semantically deep embeddings from the RoBERTa transformer with a set of carefully designed language statistics and linguistic statistics and shows excellent resistance to the surface-level adversarial paraphrasing strategy.
Anita Rani, Suman· International Journal of Sci...· 0 citations
A new hybrid Approach for CLPD is proposed, which combines semantic information from WordNet with the syntactic structure from Universal Dependencies, then these relations are modeled in knowledge graphs for multiple language pairs, demonstrating clear improvements over state-of-the-art baselines.
Chaimaa Bouaine, F. Benabbou, Amine Bouaine et al.· IEEE Access· 0 citations
Public procurement involves the allocation of substantial financial resources; therefore, continuous oversight through audits, controls, and monitoring mechanisms is essential. However, stakeholder comments and publicly available government data are often underutilized, despite their potential to reveal procedural irregularities. To address this gap, this paper analyzes metadata from Ecuador's Sistema Oficial de Contrataci\'on P\'ublica (SOCE, Official Public Procurement System), with particular emphasis on participant comments generated during the pre-contractual phase. We propose a hybrid modeling framework that integrates unsupervised clustering and supervised classification within a natural language processing (NLP) pipeline to uncover latent patterns and detect potentially irregular procurement processes. Semantic embeddings are generated using Word2Vec, LLaMA, and RoBERTa, followed by Gaussian Mixture Models (GMMs) for unsupervised clustering. A supervised classification stage is then applied to identify accusatory or whistleblowing-style comments. Experimental results show that the combination of domain-trained Word2Vec embeddings, GMM-based clustering, and a Random Forest classifier achieves high precision and recall, even under severe class imbalance. These findings demonstrate that lightweight, domain-adapted NLP architectures can effectively support risk identification and enhance transparency in public procurement systems without requiring large-scale computational infrastructure.
Bryan Torres, Daniel Riofrío, J. Vega-Sánchez et al.· 0 citations
The rapid advancement of generative Artificial Intelligence (AI) presents challenges to academic integrity due to potential misuse like plagiarism. This study develops a text detection system specifically for the Indonesian language using a Deep Learning approach with a Bidirectional Long Short-Term Memory (Bi-LSTM) architecture. The research methodology follows the Cross-Industry Standard Process for Data Mining (CRISP-DM) framework. A dataset comprising 5,008 text rows was compiled via web scraping from journalism platforms and academic journals indexed in SINTA 4 for human-written texts, while AI-generated counterparts were engineered using ChatGPT and Google Gemini paraphrases. Text features were extracted using a Keras Tokenizer and Embedding Layer with 64 dimensions. Evaluation of the trained Bi-LSTM model on a 30% validation split demonstrated an overall accuracy of 78.24% and a Mean Absolute Error (MAE) of 0.3295. Specifically, the model achieved a 93.77% success rate in identifying human-written texts, though it logged a lower detection rate of 62.62% for academic AI text structures. The final model was successfully deployed as a web application using Streamlit.
Rafil Moehamad Alif, Syariful Alam, Chandra Dewi Lestari· Jurnal Teknologi Informatika...· 0 citations
The rapid proliferation of digitally distributed news has made large-scale automated verification an urgent research priority, as deceptive content spreads across social platforms faster than manual reviewers can evaluate it. This work introduces a two-stage deep learning pipeline in which a pretrained BERT encoder generates rich contextual token embeddings that are subsequently processed by a Bidirectional Long Short-Term Memory (BiLSTM) network, enabling the classifier to exploit both sentence-level semantics and document-level narrative flow simultaneously. Experiments on the WELFake benchmark [1] following a standardised cleaning pipeline—lowercasing, URL stripping, tokenisation, stop-word removal, and WordNet lemmatisation— yielded 98.7 % accuracy and an F1-score of 0.986. This surpasses a BERT-only baseline by 3.1 percentage points and exceeds several previously published state-of-the-art results. The gains confirm that coupling transformer-based semantic representations with recurrent sequential modelling produces a measurable and reproducible improvement in misinformation detection. Planned extensions include multilingual evaluation and knowledge-distilled encoder replacements targeting real-time throughput constraints.
Ardra P Namboodiri, Archa P S, Honey Mol O· 2026 6th International Confe...· 0 citations
AssistEM, a framework for efficient LLM adaptation to EM via principled data selection, demonstrates that selective fine-tuning not only accelerates adaptation but also improves training efficiency (requiring fewer GPU hours), enabling open-source LLMs to rival–and in some cases outperform–closed-source models.
John Bosco Mugeni, S. Lynden, Toshiyuki Amagasa et al.· International Journal of Dat...· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduAug 18, 2026
A new method for surgically removing training examples from a model reveals that as datasets grow, the link between what a model learns and what it produces dissolves.
