BLR-850: 850 Blocksworld, 850 Logistics and 850 Rovers PDDL Problems and Natural Language Prompts

Abstract

Benchmark corpus for the manuscript "Grounding Large Language Models in a Symbolic Formalization Environment for Natural-Language Planning". The corpus contains 2,550 natural-language planning requests with their ground-truth PDDL problems. It covers 850 problems in each of three International Planning Competition (IPC) domains, which form a graded progression in formalization demand. Blocksworld requires only binary relations between objects. Logistics adds object categories and a spatial, hierarchical structure expressed through predicates of arity greater than two. Rovers requires typed objects with properties, numeric fluents for traverse distances, and an optimization metric. It is the IPC 2005 numeric variant, without its temporal aspect. How the data was created. Blocksworld and Logistics problems were generated with the generator distributed with PlanBench, and Rovers problems with the IPC 2005 generator. Problem size is defined as the number of objects, initial conditions, goals and metrics in the ground-truth problem. For Logistics and Rovers, this size replaced solution difficulty as the generators' sampling criterion, and in Blocksworld the size level follows the number of blocks (4 to 20). Each domain is stratified into 17 size levels of 50 problems each. Hashing confirmed that no two problems coincide, and degenerate problems, whose goals were tautological or already satisfied in the initial state, were discarded. The natural-language requests for Blocksworld and Logistics were produced with PlanBench's templated generator, keeping only the text after its STATEMENT marker. Each request therefore states an initial situation and a goal in an "I have / I want" format, and contains neither domain information nor examples. The same method was adapted to produce more natural-sounding Rovers requests. Object names in a request may differ from those in the ground-truth problem, so formalizations should be compared up to object renaming. Contents: blocksworld/, logistics/, rovers/: each folder contains the domain PDDL file, ground_truth/ with 850 PDDL problems (instance-1.pddl to instance-850.pddl), and prompts.json with the natural-language request for each instance. instances.csv: the domain, instance number, ground-truth file, size level and declaration count of every instance. The Rovers declaration counts include some initial conditions that the IPC generator writes twice, as the stratification did. LICENSE: Creative Commons Attribution 4.0 International (CC BY 4.0).