Trust by design in AI-augmented procurement systems: the roles of explainability, governance, and human oversight

Abstract

With the growing integration of artificial intelligence (AI) into buyer–supplier negotiations, procurement teams must translate efficiency gains into defensible and appropriately calibrated reliance on AI-mediated decision support. This study develops and empirically evaluates a socio-technical trust-by-design model for AI-augmented procurement negotiation systems. It jointly considers perceived transparency/explainability (XAI), ethical governance visibility, and human-in-the-loop (HIL) relational design as conceptually distinct levers of AI system trust—that is, professionals’ willingness to rely on an AI negotiation system and their belief that it behaves competently and responsibly. AI system trust is treated as a focal proximal outcome, not as a proxy for, or a sufficient explanation of, dyadic buyer–supplier trust. Using an exploratory sequential mixed-methods approach—six multi-sector case studies across the EU and ASEAN regions (36 interviews) followed by a purposively recruited cross-sectional survey of 238 AI-exposed professionals analyzed via structural equation modeling (SEM)—we find that AI use is associated with higher perceived negotiation efficiency, but also with lower AI system trust when automation displaces relational cues and explanations are weak. XAI exhibits the strongest positive association with AI system trust, while HIL design and governance show additional positive associations. In an augmented specification, XAI and HIL statistically account for the AI use–trust association. A small inverse-U pattern further suggests that moderate, well-governed AI use is associated with higher trust than very low or very high automation intensity. Multi-group analyses indicate stronger XAI–trust associations in the EU and stronger HIL associations in ASEAN. These findings contribute to trust-in-AI and procurement research by (i) clarifying the boundary between trust in an AI system and trust in a buyer or supplier, (ii) specifying procurement-specific conditions—confidentiality, auditability, and negotiation tacticity—under which AI use can erode confidence in AI support, and (iii) offering a cautiously framed, evidence-consistent roadmap for trustworthy AI negotiation systems. The reported associations should be interpreted within a non-probability sample and do not establish population-representative effects or buyer–supplier relationship outcomes.