LLM-Text2KG: Um Pipeline Orientado por Ontologias para Construção e Avaliação de Grafos de Conhecimento Extraídos de Documentos Institucionais

Abstract

Institutional documents, such as meeting minutes, contain valuable organizational knowledge, but their unstructured nature hinders systematic analysis. This work presents LLM-Text2KG, an ontology-guided pipeline for constructing knowledge graphs from institutional documents. The pipeline was evaluated on ten meeting minutes from the Acaraú River Basin Committee using six large language models under zero-shot and few-shot strategies. The evaluation combined Precision, Recall, and F1-score against a manually annotated gold standard, ontology-conformity validation, competency questions expressed as SPARQL queries, and an ablation of the ontology representation. Gemma 4 31B in few-shot achieved the best performance, with F1-score of 0.650 for entities and 0.498 for relations, and 99.37% ontology conformity. The competency questions achieved 41.0% overall coverage. Replacing the Turtle ontology with a structured textual list reduced relation F1-score from 0.464 to 0.102 in zero-shot and from 0.498 to 0.265 in few-shot. The results indicate that ontology representation influences extraction quality and that semantic relation extraction remains the main challenge.