Skip to content

Retrieve, Match, Escalate: Accurate and Scalable Product Linking with VLM-Distilled Cross-Encoders and Agentic VLMs

Aug 2026 · 0 citations · 28 references
Computer Science

TL;DR

This work presents a production retrieve-then-match cascade that spends computation in proportion to difficulty: retrieval surfaces plausible matches, a lightweight text cross-encoder auto-resolves the high-confidence majority, and an agentic multimodal vision-language model settles the ambiguous remainder.

Abstract

Product linking, the entity-resolution task of mapping merchant product records to canonical catalog products, consolidates fragmented listings so downstream search, recommendation, and advertising see one clean entry per product. At marketplace scale, billions of noisy, multi-category records must be resolved against tens of millions of canonical products, where scoring every candidate with a single model is either too weak for the hard cases or too costly for the easy ones. We present a production retrieve-then-match cascade that spends computation in proportion to difficulty: retrieval surfaces plausible matches, a lightweight text cross-encoder auto-resolves the high-confidence majority, and an agentic multimodal vision-language model settles the ambiguous remainder by inspecting product images and issuing web searches for evidence that is in neither record. The cross-encoder is distilled from millions of dual-VLM-consensus labels, retiring human annotation from the training set, and is calibrated to auto-accept links at a 98% precision bar validated against a smaller operator-certified audit. The agent is a self-hosted open-weight model that reaches a closed frontier VLM's precision at a four-point recall cost (88% versus 92%) for roughly one-seventh the per-pair cost, with no fine-tuning. Per-pair cost spans nearly five orders of magnitude from the cheap cross-encoder to the frontier VLM, so escalating only the hard tail to the agent raises end-to-end link coverage from the cheap stage's 68% to 77%.

View source

Similar papers

#small language model Open access Aug 2026

PARA: Perception, Action, Reasoning, Adaptation. Four Faculties an Institution Can Revoke

The fourth faculty is Adaptation. Any source rendering it as Reflection is in error, including sources by this author, and the distinction is not cosmetic: reflection is a private act with no external consequence, while adaptation writes to institutional memory, which is why it needs a guardrail and why misnaming it removes the reason for one. No trademark is claimed on PARA or on any of the four faculty names. The construct is offered for use, teaching, assessment, extension and criticism by anyone, with attribution, under CC BY 4.0. An operational agent that watches a system and acts on it is usually described as a perceive-and-act loop, and the description omits the two things an institution needs. It omits the reasoning that justifies an action, which is the only part that can be argued with once the action turns out to have been wrong. And it omits the adaptation that closes the loop, which is where the agent's experience becomes something the institution keeps. PARA names four faculties, each carrying a distinct authority type. Perception has read-only access to system signals and emits structured observations, distinguishing what was measured from what was inferred. Reasoning has read access to observations and runbooks, emits a plan and its justification, and writes nothing at all, which is what makes it safe to give it the widest read access of the four. Action holds the sole authority to change production, through enumerated policy-authorized operations only. Adaptation has write access to institutional knowledge and no write access to production. Two faculties write and two do not, and the two that write are the two that carry guardrails. The substantive requirement is that Adaptation is bounded by the same guardrails as Action, which reads as excessive until the failure it prevents is named. An agent that could both act and rewrite the record of its action could launder its own mistakes into institutional memory, and the institution would then improve its future decisions from a corrected account. Nothing about that is detectable downstream, because the record is the only thing downstream has and there is no second copy to compare against. The failure does not require a deceptive agent: one adapting honestly from a mistaken belief about its own action produces the same result, which makes the guardrail a defence against a normal agent rather than a malicious one. The second requirement is the registry entry that turns a faculty from a description into a contract, carrying the faculty, its allowed actions, its forbidden actions, its governing guardrail and its success metrics. Forbidden actions are named although they are formally the complement of the allowed set, because a reviewer cannot otherwise tell a capability deliberately withheld from one nobody thought of. Success metrics sit in the same entry because the metric is what the agent's optimizer pushes against the guardrail. An agent must not exercise a faculty its entry does not record, and an agent that quietly acquires one usually does so incrementally and with good intent: a reasoning faculty given a small write to make itself useful is an action faculty with no guardrail. The acronym and the loop are in different orders, which the specification states explicitly because the mismatch is a reliable source of confusion. The acronym reads P-A-R-A; the loop runs perception, reasoning, action, adaptation, and reasoning precedes action so that a justification is not constructed afterwards. This is the depth treatment of pattern OP-5 of A Pattern Language for Production LLM Platforms, which is the canonical statement and governs where the two disagree. Documented uses of the full four-part model are emerging rather than established, no implementation unconnected to the author has been evaluated, and the laundering failure is argued rather than observed, which the specification records as a weakness of the argument and not only of the phenomenon. It is a specification, not a certification scheme.

Nabeel A. Khan · 2 citations
#small language model Open access Aug 2026

LifeSciBench: Evaluating Language Models on Realistic, Expert-Level Tasks in the Life Sciences

LifeSciBench is introduced, a benchmark of 750 expert-authored tasks designed to evaluate whether language models can handle realistic life science research work, with each constituent task paired with a human expert-written rubric.

Amelia Liu, Andrew Ho, Anne Marie Droste et al. · 2 citations
#small language model Preprint Aug 2026

HEPToolBench 1.2: Testing How Reliably Language Models Can Drive Particle Physics Software

HEPToolBench is introduced, a benchmark of 28 collider-simulation tasks scored by deterministic, task-specific scorers, plus a three-task structured-debugging extension, and moving syntax generation into deterministic software can substantially improve reliability for both small local and frontier models.

Unknown authors · 1 citation
#small language model Preprint Aug 2026

CritICL: Inference-Time Weak-to-Strong Generalization from Small Language Model Failure Modes

Experimental results show that CritICL consistently outperforms standard in-context learning and achieves performance competitive with or superior to test-time scaling methods, while requiring significantly fewer generations and lower token cost.

Yu-Fan Wu, Yinghui He, Zhengyi Hu et al. · 1 citation

Low Carbon Scheduling of Integrated Energy System Based on Large Language Model-Embedded Multi-Agent Reinforcement Learning

The complex multi-energy coupling characteristics inherent to integrated energy system (IES) present unprecedented challenges for the implementation of low-carbon scheduling. Existing optimization methods often exhibit limitations in system scalability, algorithm adaptivity, and carbon reduction efficacy for complex IES. This paper proposes a Large Language Model (LLM)-Embedded Multi-Agent Reinforcement Learning (LEMARL) to address the aforementioned issues. The proposed method integrates the global perception capability of LLMs with the dynamic optimization capability of MARL. Specifically, the LLM-Embedded module generates high-quality reward functions and policy frameworks from a global perspective, while the MARL module leverages these LLM-generated strategies for distributed interactive iterations—greatly enhancing computation efficiency and scalability. Simulation results demonstrate that LEMARL reduces carbon emissions by 7.76% and simultaneously decreases operating costs by 4.49% in a small-scale IES. Furthermore, LEMARL also exhibits superior applicability and scalability in large-scale IES of the IEEE 141-bus power grid integrated with 51-node thermal system.

Chen Xia, Tong Gou, Yinliang Xu et al. · 1 citation

Related blog posts

Microsoft Research Blog Aug 31, 2026

GigaPath-Flash and GigaTIME-Flash: Toward population-scale discovery with efficient pathology foundation models

What if pathology foundation models could do more with less? GigaPath-Flash and GigaTIME-Flash cut computational demands while maintaining strong performance, opening the door to larger studies and broader exploration. The post GigaPath-Flash and GigaTIME-Flash: Toward population-scale discovery with efficient pathology foundation models appeared first on Microsoft Research.