Skip to content
Open access

LLM-MetaAS: A Semantic-Statistical Policy Routing Framework for AutoML Execution Strategy Selection

Jul 2026 · Applied Sciences · Vol 16, pp. 7033 · 0 citations · 18 references

TL;DR

LLM-MetaAS is proposed, a semantic-statistical framework for AutoML execution policy selection that improves overall AutoML performance and selects policies closer to the oracle than fixed strategies, random selection, and the native Auto-sklearn 2.0 selector.

Abstract

Automated machine learning (AutoML) aims to reduce manual intervention in machine learning pipeline construction. However, AutoML performance is affected not only by the searched pipeline but also by high-level execution policies, such as validation protocols and budget allocation strategies. Auto-sklearn 2.0 has shown that these policies can be selected automatically, but its lightweight task representation mainly captures dataset scale and may not distinguish tasks with similar sizes but different semantic and structural characteristics. To address this limitation, this article proposes LLM-MetaAS, a semantic-statistical framework for AutoML execution policy selection. Building on the established pairwise policy-selection paradigm, LLM-MetaAS focuses on improving task representation rather than introducing pairwise decomposition itself. It constructs a policy-oriented fingerprint by combining LLM-assisted semantic profiling with lightweight statistical descriptors related to validation reliability and computational demand. Policy quality is evaluated using an explicit regret-based criterion relative to the empirical oracle, and vote margins are used to analyze routing uncertainty. Experiments on 39 benchmark tabular classification datasets show that LLM-MetaAS improves overall AutoML performance and selects policies closer to the oracle than fixed strategies, random selection, and the native Auto-sklearn 2.0 selector. Ablation and robustness analyses further support the utility of the complete semantic-statistical representation within the evaluated framework.

Read PDF

Similar papers

Book Open access Jul 2026

LLM Policy Induction for Heuristic Search Control: A Trace-Driven ALNS Framework

Experimental results demonstrate that the proposed framework consistently outperforms classical ALNS baselines and several competitive metaheuristic methods in terms of solution quality and convergence speed.

T. M. Nguyen · 0 citations
Open access Jul 2026

LLM-Advisor: Dynamic Model Selection and Query Routing in Heterogeneous Multi-LLM Architectures

The rapid proliferation of Large Language Models (LLMs) with varying capability profiles, context window limits, execution latencies, and financial costs presents a significant operational challenge for enterprise AI deployments. Monolithic deployment strategies wherein all requests are directed to a single high-capability frontier model result in substantial compute over-provisioning and excessive operational costs for routine queries. Conversely, relying solely on lightweight models degrades output accuracy on complex multi-step reasoning tasks. To resolve this trade-off, this paper introduces LLM-Advisor, an open-source, adaptive framework designed for intelligent query categorization, dynamic model evaluation, and constraint-aware request routing across heterogeneous multi-LLM pools. LLM-Advisor analyzes incoming prompt features, structural complexity, domain requirements, and user-defined constraints (e.g., maximum cost per request, latency thresholds) to route tasks to the optimal candidate model. We evaluate LLM-Advisor using a benchmark suite of 1,000 queries across code generation, general reasoning, and contextual retrieval tasks using both proprietary and open-weight models (including GPT-4o, Claude 3.5 Sonnet, Llama 3, and Mistral). Experimental results demonstrate that LLM-Advisor achieves a 42% reduction in overall inference expenditure and a 35% decrease in average response latency while retaining 94.6% task accuracy compared to static GPT-4o baseline routing. These findings highlight LLM-Advisor as an efficient, highly scalable middleware solution for production-grade AI system deployments.

Harshil Lodhiya · 0 citations
Open access 2026

TabGuard: Agentic LLM Orchestration for Adaptive Tabular Anomaly Detection via Dynamic Validator Selection and Generation

Tabular anomaly detection is challenging because real-world tables contain heterogeneous columns, ranging from structured identifiers to free-form text. Existing methods face a fundamental trilemma: rule-based systems require extensive manual configuration and fail on novel schemas; statistical methods scale efficiently but miss semantic errors; and LLM-based approaches understand semantics but incur prohibitive per-cell inference costs. No prior method simultaneously addresses semantic heterogeneity, domain-specific validation rules, and enterprise-scale processing. We introduce T AB G UARD , an agentic framework that resolves this trilemma through semantic routing. Using LLM function calling, the system analyzes a small sample of each column and dynamically selects the most effective validation strategy, routing to a regex-based valida-tor for syntactic patterns, a code-generation val-idator for domain-specific rules (such as Luhn checksums for credit cards), or an embedding-based validator for distributional outliers. This architecture decouples expensive cognitive reasoning ( O ( m ) LLM calls for m columns) from scalable programmatic execution, enabling deployment on enterprise datasets without per-cell inference.

Srihari Unnikrishnan, Minghuan Ma · 0 citations
Preprint Aug 2026

AutoSND: From Execution Evidence to Structural Policies for Automated Network Dismantling Heuristic Discovery

Network dismantling is fundamental to analyzing the robustness and vulnerability of complex systems, yet practical heuristics must balance effectiveness and computational efficiency, and are usually designed manually by researchers. Existing large language model based automatic heuristic design methods can generate and screen candidates, yet they have difficulty further transforming candidate quality or failure states during execution into structural-level guid- ance for subsequent generation. We propose AutoSND, a three stage tree search framework for complete network dismantling pro- grams. Stage I broadly explores from simple heuristics and archives execution evidence. Stage II compiles candidate records into struc- tural policies concerning local signals, neighborhood access, and state update ranges. Stage III continues tree search conditioned on these policies and obtains the final quality prioritized and speed prioritized candidates, AutoSND-Q/S. Experiments on 12 real world networks and 3 large real world networks show that AutoSND achieves better search performance and stability and discovers more competitive and structurally interpretable network disman- tling programs. The final candidates form an interpretable structure that uses residual degree as the backbone, adjusts node order with bounded local signals, and restricts the state update range. Code is available at https://github.com/MirrorNew/AutoSND.

Zhijing Hu, Changjun Fan, Yufan Deng et al. · 0 citations
Preprint Jul 2026

MetaRoute-Bench: Evaluating Meta-Decision Policies for Agentic Workflow Routing

Agentic systems must repeatedly decide whether to answer directly, decompose a task, invoke a tool, execute code, delegate to a specialist, verify an intermediate result, or recover from failure. These meta-decisions affect not only task success but also operating cost and latency, yet they are often embedded inside an orchestration framework and evaluated only through aggregate task accuracy. We present MetaRoute-Bench, an open, inspectable framework for comparing meta-decision policies under a shared execution model. The initial benchmark contains 180 synthetic task profiles spanning data analysis, research, and document processing, eight routing policies, and 30 paired random seeds. Across 43,200 traces, a task-aware compositional policy achieves 79.4% success compared with 76.7% for a strong workload-specific static policy, 67.4% for one-shot task routing, and 52.9% for direct answering. Relative to the static policy, this is a 2.7 percentage-point improvement with paired 95% CI of plus or minus 2.0 points, at 4.7% higher mean cost and 6.4% higher latency. Ablations show the largest losses when route composition is restricted to one operation and when verification is removed. These results are generated by a seeded offline execution model rather than a live deployment; accordingly, the primary contribution is a reproducible evaluation method and an analysis of routing-policy tradeoffs, not evidence of production effectiveness. We release task generation, policies, traces, tests, and analysis artifacts to support live-system validation.

Natan Vidra, Alina Kapanova, Arun Kanhai et al. · 0 citations