Improving LLM-based event extraction with annotation guidelines

Abstract

Event extraction constitutes a foundational task in information extraction, but reliance on laborious and expensive human annotations severely restricts the availability of training datasets. While recent works have explored Large Language Models (LLMs) as example-driven (or zero-shot) annotators, they are substantially outperformed by supervised techniques on structured extraction tasks, such as event detection and argument extraction, possibly on account of underspecified task instructions. In this work, we investigate to which extent LLMs can benefit from dataset-specific, detailed annotation guidelines that more precisely represent the dataset's underlying (human) annotation procedures. To this end, we propose a guideline-based, three-stage LLM annotation framework for event extraction that incorporates detailed event annotation guidelines and supports multiple LLM annotators to improve robustness. Using the comprehensive and well-documented ACE 2005 English Annotation Guidelines for Events as a reference document, we evaluate four LLMs across three guideline-compliant benchmark datasets. Our findings indicate that, depending on model choice, guideline specificity, and the dataset's relative label accuracy, employing detailed guidelines can considerably boost event extraction performance, gaining up to 6.7 F1 points over commonly used bare-minimum instructions, with particularly remarkable improvements for reasoning-based models. Furthermore, we demonstrate that augmenting existing datasets with LLM-generated argument annotations can improve argument extraction performance under soft-matching evaluation. Overall, our experiments emphasize the importance of annotation guidelines (as well as their specificity) for LLM-based annotations, providing valuable insights on leveraging LLMs as guideline-compliant annotators.