AgentSearch: Indexing, Retrieval, and Ranking of AI Agents

Abstract

As AI agents and tools increasingly carry out everyday tasks on behalf of users, the digital ecosystem is rapidly filling with agents that offer overlapping or complementary functionalities, raising a fundamental challenge: how can users, developers, and orchestrating systems effectively identify, compare, and select the most appropriate agents or tools for a given task? While rooted in traditional information retrieval (IR), agent search introduces new challenges because the retrieved entities are executable systems rather than static information artifacts, and their suitability depends on capabilities, behaviors, constraints, uncertainty, and task-dependent performance. We argue that the IR community is well-positioned to advance this emerging problem setting, which we refer to as AgentSearch, by developing principled methods for representing, indexing, retrieving, and ranking AI agents and tools under multi-dimensional relevance criteria. The AgentSearch Workshop aims to bring together researchers and practitioners from academia and industry to explore the challenges and opportunities of agent and tool search, including agent discovery, capability representation, retrieval and ranking models, evaluation methodologies, and issues related to safety, fairness, personalization, and explainability. To support concrete progress, the workshop will also host an exploratory AgentSearch Challenge that provides a shared experimental setting for ranking agents given a task. The workshop will adopt an interactive format featuring breakout discussions, poster and spotlight sessions, invited talks, and collaborative activities, fostering active engagement and cross-disciplinary exchange beyond a traditional mini-conference structure.