GeoAct: evaluation materials for action-level access control in a geospatial LLM agent

Abstract

Evaluation materials for the paper "GeoAct: action-level access control for agentic geospatial analysis with local large language models". GeoAct is a geospatial agent that plans and executes spatial-analysis tool calls on behalf of a user in a multi-tenant spatial data infrastructure. Every tool invocation is authorised against the user's dataset-level permissions before the tool reaches storage, by a deterministic gate called the Action-Level Access-Control Enforcement Point (Action-PEP). This deposit holds the mechanism, the evaluation behind the paper, and the generators for every table and data figure in it. Contents: - code/tools.ts - the read-only GIS tool set and the dataset arguments each tool declares, which is what the Action-PEP gates. Contains the gate itself.- code/agent.ts - the Planner-Action-PEP-Executor loop. Every tool call is authorised before dispatch; denials are counted and audited.- code/geoact_bench.ts - GeoAct-Bench: builds the per-tenant plans, generates the legitimate and adversarial requests together with their ground truth, runs every condition over every planner model, and writes the results and tables.- code/geoact_conformance.ts - the gate-level harness, which needs no language model. Arm A sweeps every (tool, dataset-argument, user, dataset) combination through the enforcement path and checks both directions, unsound execution and over-blocking. Arm B injects plausible defects - bypass mutants that make the gate reach fewer arguments than it should, and policy mutants that weaken its access predicate by one disjunct - and repeats the sweep unchanged.- code/geoact_testbed_export.ts - exports the study-area geometry, transforming the stored WGS84 extents to ETRS89 / TM35FIN in PostGIS.- data/geoact.json - request-level results: unauthorised-access counts and rates per model and condition, content-leak counts, task success, mean steps, mean latency, gate overhead, and the catalogue composition.- data/geoact_conformance.json - gate-level results: per-tool conformance counts with Wilson intervals, and the outcome of every mutant.- data/testbed_extents.json - one projected bounding box per site and tenant. Tenants are anonymised to letters; no dataset names, owners or contents are included.- tables/ - the result tables as they appear in the paper, and make_tables.py, which regenerates all of them from the two result files.- figures/ - make_gate_spectrum.py and make_testbed_map.py, which regenerate the gate-spectrum figure and the study-area map from the data files. Scope and limits: the Action-PEP itself, which is the object of the paper's claim, is included in full, so the gate's placement and logic can be audited directly, and every number in the paper's tables and data figures derives from the files in data/. The conformance and mutation arms are deterministic and need no language model. They still require the live catalogue for the permission relationships, so they cannot be run from this deposit alone, but they involve no sampling: given the same catalogue they return the same counts. The mutation logic can be audited against tools.ts directly, since what it manipulates is the datasetArgs declaration that the Action-PEP reads. A synthetic catalogue is included so that the evaluation can be re-run from this deposit alone. fixture/ holds wholly invented datasets, sites and tenants reproducing the structure the benchmark depends on - disjoint private holdings per tenant, sites where two tenants hold different products over the same ground, a few published datasets - and, unlike the live catalogue, access granted through both the per-user and the per-group route. Load it into any PostGIS database and point the harness at it. Against the fixture all eleven mutants are detected, where the live catalogue detects nine, so the limitation reported in the paper belongs to the testbed rather than to the harness. No real dataset, site, holding or person is represented. The benchmark cannot be re-run against the live catalogue. The harness queries a live geoportal database for the catalogue, its geometries and its permission relationships, and drives a locally served language model; neither is included, because the catalogue contains access-controlled datasets belonging to identifiable researchers and the permission structure between them is the object of the study. The two arms were run on different catalogue snapshots. The request-level benchmark reports the July 2026 catalogue; the conformance and mutation sweeps were run later. Both snapshots are recorded in the respective JSON files. The request-level results are aggregates rather than per-request records. The harness retains counts and rates per model and condition, not the individual adversarial requests, the tool calls the planner emitted, or the agent's answers. Licensing: code under code/ is MIT. The data, tables and figure generators are CC BY 4.0.