Instrumentation vs. Observability: Provenance Capture Trade-offs in Agentic Workflows
Abstract
Scientific Multi-Agent Systems (SMAS) increasingly execute scientific agentic workflows, where AI agents coordinate tasks, invoke tools, exchange information, and make runtime decisions. However, a reasoning error produced by one agent can silently become another agent’s input, propagating and compounding through the workflow until it corrupts scientific conclusions or misdirects costly experiments. Thus, tracing how each agent’s outputs shaped downstream decisions is essential for reliability, debugging, accountability, and reproducibility. While provenance has long supported scientific workflows, its integration into SMAS remains unclear. Provenance can be captured through code instrumentation or by observing data and execution artifacts, with each approach affecting capture scope, intrusiveness, system dependence, domain-awareness, and analytical capabilities. This paper characterizes and analyzes six capture strategies across two categories, instantiated using Academy and Flowcept as example agentic framework and provenance system, respectively, on a Perceptron Grid Search SMAS workflow. Results show that all strategies capture core workflow provenance, but none captures it fully; they differ in representing agent semantics, resource usage, and platform context, suggesting that combining complementary strategies can maximize overall coverage.