Skip to content

WinSyn: An Automated Pipeline for Realistic Enterprise Question-Answering Evaluation

Sep 2026 · 0 citations · 32 references
Computer Science

TL;DR

This work introduces an automated pipeline for generating synthetic datasets of emails reflecting realistic workplace scenarios, along with long- and short-form questions and gold answers grounded in the data, and underscores the importance of realistic, high-complexity evaluation data for developing stronger real-world enterprise DR systems.

Abstract

Enterprise settings provide a challenging environment for question-answering agents, which often rely on Retrieval-Augmented Generation, Deep Research (DR), and related techniques. Much of this challenge comes from the complexity of enterprise data: information is often spread across evolving and potentially conflict- ing emails, chat messages, documents, and other artifacts. Existing benchmarks typically have limited real-world complexity, short-form responses, and unnatural queries, so they often fail to capture the challenges of enterprise settings. In this work, we introduce an automated pipeline for generating synthetic datasets of emails reflecting realistic workplace scenarios, along with long- and short-form questions and gold answers grounded in the data. Our method simulates long-running enterprise projects spanning several months and involving up to 25 interacting employees across multiple roles. The data emphasizes ambiguity, distributed information, and naturally occurring queries. To validate the pipeline, we evaluate few standard agentic baselines on our datasets using the latest frontier models. We find that aggregate scores averaged over all queries remain below 80% for each dataset, indicating significant room for improvement. These findings suggest that more work remains to be done for enterprise deployment and underscore the importance of realistic, high-complexity evaluation data for developing stronger real-world enterprise DR systems.

View source

Similar papers

Open access 2026

Enhancing and Evaluating Tabular Models on the Fly via Synthetic Question-Answer Generation

S YN T AB QA defines a detailed question typology, enabling fine-grained evaluation and facilitating the creation of diverse QA datasets, and not only provides an automated test bed for any tabular dataset but can be used in few-shot settings to supply LLMs with tailored examples, improving their focus and accuracy.

Jorge Osés Grijalba, Eugenio Martínez-Cámara, Luis Alfonso Ureña López et al. · 0 citations
#artificial intelligence Preprint Sep 2026

DocHop: Benchmarking Out-of-domain Multi-hop Reasoning in Information-Dense Documents

This work introduces DocHop, a benchmark for integrated chart--context reasoning in document-style images and constructs DocHop via a stochastic logic-first generation pipeline with controllable reasoning depth and visual density, to enable systematic evaluation.

Zhuo-Ran Yu, Le Thien Phuc Nguyen, Jaden Park et al. · 1 citation

NoteVQA: Benchmarking VLMs on Real-Life Questions from Human Communities

This work introduces AgenticInterleave, a single-agent ReAct framework for retrieval-supported answer generation, together with IVR-12, a 12-dimensional rubric for assessing the content, presentation, and image quality of interleaved references and model outputs, and evaluates both short-answer correctness and interlea...

Hao-Nan Jiang, Guo-Jian Zhan, Jian-Cong Xie et al. · 0 citations
#machine learning Open access Oct 2026

FALCON: A Model and Dataset Agnostic Framework for Synthetic Data Generation for NL2SQL Pairs

FALCON is presented, a framework that generates realistic, ambiguity-aware NL-to-SQL data matching the complexity of challenging real-world benchmarks, at low cost using compact open models and the model- and database-agnostic design enables organizations to generate high-complexity NL-to-SQL training data locally with...

Darian Lee, Shannon Rumsey, J. St. Clair et al. · 0 citations
#artificial intelligence Preprint Sep 2026

CruxBench: A Benchmark of Information Discovery

CruxBench is introduced, a benchmark that grades LLM-generated questions by their Value of Information (VOI): how much a model-proposed crux updates beliefs about a target forecasting question and finds that VOI correlates highly with independent measures of model capability and captures cruxes'usefulness for answering...

Hui Dai, Li-Na Piao, Nick Merrill et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.