2026· Zbornik Veleučilišta u Rijeci· Vol 14, pp. 279-295· 0 citations· 54 references
TL;DR
Comparisons of large language models with a system built using the Node of Knowledge conceptual framework revealed similarities and differences between the systems, which are presented in this paper.
Abstract
A system based on the conceptual framework Node of Knowledge (NOK) enables the recording ofnatural language sentences and questions in the NOK relational database, as well as the retrievalof answers to those questions. Large Language Models (LLMs) are designed for the same purpose:to provide answers to questions. Therefore, comparing large language models with a system builtusing the Node of Knowledge conceptual framework is an important research question. This paperaddresses this by comparing these two systems. Among large language models, GPT was selectedfor comparison as it is one of the most widely used models. The comparison was conducted in twoparts. First, the process models of the two systems were compared. Second, an analysis was performedon the answers produced by the NOK-based system and by ChatGPT, representing large languagemodels, for a selected natural language sentence and set of questions. All comparison elementsrevealed similarities and differences between the systems, which are presented in this paper.
Decisions are being made based on data, and the demands for easy-to-use interfaces that enable users to simply type in a question to a relational database without having to be a Structured Query Language (SQL) expert have increased. Although the development of Large Language Models (LLMs) has seen remarkable advancements in the past few years, and has led to the emergence of LLMs that have significantly improved the generation of SQL from natural language (Text-to-SQL)[1], there is a relative lack of systematic comparisons between the latest commercial LLMs that has been tested against a series of graded SQL complexity levels with both syntactic and execution-based metrics. This research aims to assess and contrast ChatGPT, Google Gemini, DeepSeek, and Microsoft Copilot for their accuracy and efficacy in creating SQL queries from natural language queries. In a quantitative experimental design, 100 natural language questions were given using a purpose-built benchmark, with six types of queries that included simple retrieval, filtering, joins, aggregation, GROUP BY and HAVING, and nested subqueries. The results of the models were assessed using the manually written and validated ground truth query, with the use of normalised Exact Match Accuracy (EMA) and Execution Accuracy (EXA). The best overall results (EMA 97%, EXA 97%) were obtained by ChatGPT. The Copilot scored EMA 56% and EXA 96%, DeepSeek EMA 55% and EXA 93% and Gemini the lowest with a score of EMA 36% and EXA 87%. The large margin between EMA and EXA for the models (+51 points, +40 points, +38 points) demonstrates the significant shortcomings of exact-match scoring when it comes to assessing real-world skills for producing valid but structurally different SQL. The study empirically demonstrates comparative behaviours of the current LLMs, and suggests the use of execution-aware evaluation in the studies and deployment of Text-to-SQL.
Bharath Kumar N, T L Manasa· International journal of com...· 0 citations
LMEnt is released to support studies of knowledge in LMs, including knowledge representations, plasticity, editing, attribution, hallucinations, and learning dynamics, finding that entity co-occurrence and mention forms—which are difficult to study with existing tools—affect learning trends.
Daniela Gottesman, Alon Gilaie-Dotan, Ido Cohen et al.· Transactions of the Associat...· 0 citations
Providing appropriate answers to questions is necessary in many situations, not just in the conversational AI systems we see and develop today. Research in Natural Language Processing (NLP) and User Modeling (UM) have investigated this topic for decades, starting in the era of ''symbolic AI''. While NLP in general was needed for the whole interaction (understanding the question and answering it), Natural Language Generation (NLG) was particularly concerned with providing good and coherent answers appropriate for the information need and the intended audience, which is where UM also played a role. At that time, information to include in the answers typically came from knowledge bases or data bases. Information Retrieval (IR) then was concerned with retrieving the documents (and later websites) most relevant to a query. As the amount of data and number of documents increased, information needs from users became increasingly complex. As a result, it seemed that combining advances in both NLP and IR was required. And of course, now, research often spans these two fields. In this talk, I will look at past work in the fields of NLG and UM, outlining what was identified as important for graceful human machine interactions. The game has changed now, of course, with LLMs and generative AI, which can do much that we could not do before. But some old questions remain unanswered, there are new questions (especially given the ''black box'' nature of LLMs), and we can probably learn from some earlier work. I will also discuss future research directions that I believe are important.
Cécile Paris· Annual International ACM SIG...· 0 citations
A “separation of concerns” (between the generative model and the structured data it consults) is proposed as a principled, declarative approach to AI-assisted XML work.
E. Beshero-Bondar, M. Wright, M. R. Simons· Balisage Series on Markup Te...· 0 citations
In recent years, large language models (LLMs) have achieved significant results in natural language processing. They are applied to various tasks, including text generation, question answering, automatic summarization, code generation, and complex reasoning. With the increasingly complex real scenarios, the length of input text that models need to deal with also grows. Thus, the long-context processing ability of language models has gradually become an important factor in evaluating the practicability of LLMs. This paper gives an introduction to the long-context processing ability of large language models. It first introduces the background of large language models and the basic concept of long-context processing. It then summarizes the main technical methods of long-context modeling, such as improving positional encoding, training stage expansion, inference-stage optimization, and architecture-level innovation. Third, the paper also discusses the use of long-context ability in long-document question answering, long-text summarization, multi-document integration, code understanding and long-context evaluation tasks. Then, summarize the current main challenges and prospects of research work. This paper argues that the ability of long context should not only come from increasing the context window, but also from the ability of the model to locate, integrate and reason about important information in long text.
Jun Wu· Applied and Computational En...· 0 citations
Automated Knowledge Base Construction (AKBC) is a core NLP task, and recent work proposes generating knowledge bases directly from large language models (LLMs), treating the model itself as the knowledge source. However, LLMs natively possess no representation of entities, leading to duplicate entries as well as conflations. We propose GPTKB 2.0, a methodology for constructing disambiguated KBs directly from LLMs. GPTKB 2.0 incorporates on-the-fly disambiguation of entities, relations and classes, and is meticulously designed to satisfy both scalability and disambiguation accuracy. We analyze the central design decisions and characterize the trade-offs between accuracy, scale, and cost. We execute GPTKB 2.0 at scale, obtaining a materialized KB containing over 1M disambiguated entities and 38.4M triples. This represents the first million-scale LLM-native KB with explicit internal canonicalization of entities, relations, and classes, a significant departure from prior Wikimedia-centric works. GPTKB 2.0 is available at https://gptkb.org/.
Yujia Hu, Tuan-Phong Nguyen, S. Razniewski· 0 citations