Data Analysis in the Wild: Benchmarking Large Language Models Against Real-World Data Complexities

D2VBench: Benchmarking Large Language Models with Value Dilemmas in Daily Scenarios

Experimental results demonstrate that D2VBench exhibits high reliability and robustness, effectively reflecting the LLMs'alignment across different value categories and dimensions, and providing a more realistic and fine-grained tool for research on value alignment.

TReB: A Comprehensive Benchmark for Evaluating Table Reasoning Capabilities of Large Language Models

The majority of data in businesses and industries is stored in tables, databases, and data warehouses. Reasoning with table-structured data poses significant challenges for large language models (LLMs) due to its hidden semantics, inherent complexity, and structured nature. One of these challenges is lacking an effective evaluation benchmark fairly reflecting the performances of LLMs on broad table reasoning abilities. In this paper, we fill in this gap by presenting a comprehensive table reasoning benchmark, TReB. Firstly, we propose a taxonomy to systematically measure both shallow table understanding abilities and deep table reasoning abilities, covering a total of 26 sub-tasks. We then construct a high quality dataset through a dedicated data processing and synthesis procedure. Based on these well-constructed samples, we design an evaluation framework to robustly measure table reasoning capabilities with three distinct inference modes. Experimental results with our data and framework reveal that existing LLMs still have significant room for improvement in addressing the complex and real world table related tasks. Both the dataset and evaluation framework are publicly available, with the dataset hosted on huggingface.co/datasets/JT-LM/JIUTIAN-TReB, and the framework on github.com/JT-LM/jiutian-treb.

Empirical benchmarking of large language models for data science coding: a multidimensional evaluation

Despite growing enthusiasm for large language models (LLMs) as coding assistants, there remains limited empirical evidence of their effectiveness in domain-specific contexts such as data science. Existing benchmarks primarily focus on general-purpose programming and do not fully capture the challenges of data science tasks, which require data manipulation, statistical reasoning, algorithmic problem solving, and visualization. They also rarely assess practical dimensions such as first-attempt reliability, output consistency, error recovery, and cost efficiency. To address this gap, we introduce the LLM4DS-Benchmark and conduct a multidimensional empirical evaluation of seven LLMs—Gemini 2.5 Pro, Claude Sonnet 4.5, o3-mini, GPT-4.1, GPT-4o, Qwen3-Coder, and Perplexity Sonar—on 814 Python data science coding problems from StrataScratch platform, spanning Analytical, Algorithm, and Visualization tasks across three difficulty levels. Each problem received up to three attempts under a branching protocol that separates independent attempts from feedback-guided retries, enabling analysis of correctness, Pass@1, retry recovery, output consistency, execution behavior, visualization quality, code similarity, token usage, and cost per solved problem. Results show that Gemini 2.5 Pro achieved the highest overall success rate (81.3%) and Pass@1 (62.2%), but at a median cost of \$0.10740 per solved problem—316 times higher than Qwen3-Coder (\$0.00034). Across models, retries improved performance by 19–24 percentage points, with feedback resolving 20–31% of initial failures. Output consistency varied significantly across identical prompts, particularly for Analytical tasks. Model rankings also shifted by task type and evaluation dimension, with no single model dominating across all categories. Instead, a Pareto-optimal set—Qwen3-Coder, GPT-4.1, o3-mini, Claude Sonnet 4.5, and Gemini 2.5 Pro—emerged, reflecting trade-offs among accuracy, cost, and reliability. These findings highlight the need for multidimensional, task-aware benchmarking and suggest that model selection for data science coding should be guided by task characteristics and practical constraints rather than aggregate success rate alone.

Automating End-to-End Hybrid Query Processing: Benchmark, Solution, and Insights

A large-scale benchmark with 60\sim 90× more queries than prior work, built on 3× more databases, an automated pipeline that can execute existing methods without manual intervention, and multi-dimensional, fine-grained evaluation metrics for comprehensive assessment.

RepBench: Compiling Benchmarks into Capability Representations for Large Language Models

Benchmarking LLM Agents on Real-World Biological Database Curation for Data-Driven Scientific Discovery

High-quality biological databases are the bedrock of data-driven scientific discovery. However, the construction of these resources remains a labor-intensive bottleneck, particularly for emerging research frontiers where structured data is non-existent. While LLM-based agents have catalyzed progress in downstream scientific modeling, their potential to automate the critical upstream challenge of database curation remains largely untapped. To bridge this gap, we introduce BioDataLab, a rigorous benchmark comprising 100 tasks meticulously derived from 57 high-impact database publications. BioDataLab evaluates the capability of autonomous agents to transform raw, heterogeneous biological resources into structured, analysis-ready databases. Unlike static evaluations, BioDataLab provides a fully interactive environment encompassing data retrieval, extraction, annotation, and integration, featuring process-oriented curation targets and contamination-control checks. We benchmark 11 state-of-the-art LLMs (including Gemini-3.0, GPT-5.2, and Claude-4.5) under different agent frameworks, revealing a substantial capability gap: the top-performing model achieves only a 40% success rate. Further error analysis identifies significant bottlenecks in multi-step tool orchestration and adherence to complex biological data formats. These findings underscore that while LLMs are proficient in downstream reasoning, autonomous upstream curation remains a formidable frontier. All data and codes are available at GitHub.