BioWeaver: Adaptive Workflow Orchestration for Biomedical Data Integration with Progressive Deep Web Exploration

Abstract

Biomedical researchers often need to combine evidence from heterogeneous sources, including REST APIs, form-based deep web databases, semi-structured web pages, and downloadable files. Existing workflow systems support reproducible analysis but require manually specified pipelines, while LLM-based agents can issue tool calls but often lack explicit data dependencies, provenance, and controlled exploration. We present BioWeaver, an adaptive workflow orchestration system that converts natural-language biomedical questions into executable scientific workflow graphs. Each graph represents typed retrieval, refinement, and synthesis steps over a catalog of heterogeneous connectors. During execution, BioWeaver uses Progressive Data Refinement (PDR) to expand the workflow when intermediate results reveal useful follow-up records, while a composite information-gain heuristic and human-in-the-loop checkpoints control exploration depth. A unified connector abstraction supports APIs, browser-automated forms, HTML tables, and downloadable files under a common execution model. We evaluate BioWeaver on 45 queries across two biomedical question sets with gold-standard answers. Results show that BioWeaver improves overall answer quality by 6.2–14.5% over LLM baselines and by 41.4–53.8% over a ReAct agent. Ablation studies show that runtime refinement, planner knowledge, and information-gain stopping each contribute to system effectiveness. These results suggest that adaptive workflow graphs provide a practical foundation for reproducible, source-grounded biomedical data integration across heterogeneous and deep web resources.