HOSPIT-LLM: A Human-Centered Multimodal Dataset and Edge-Deployed LLM Pipeline for Emotion-Aware Hospitality Assistants
TL;DR
HOSPIT-LLM provides a reproducible framework for multimodal dataset creation, preference-based fine-tuning, and deployment of human-centered AI systems, and targeted improvements in the rule-based contextual emotion-policy match and staff-emotion policy compliance are indicated.
Abstract
Large language models (LLMs) exhibit strong general conversational capabilities, yet their deployment in domain-specific service environments such as hospitality remains limited by the absence of emotionally grounded datasets and validated end-to-end system architectures. This paper presents HOSPIT-LLM, an EU-funded euROBIN Technology Exchange Program pilot, as a complete, integrated pilot pipeline for human-centered conversational AI in hotel reception scenarios. We deploy a multimodal hotel-terminal assistant in a real hotel reception, capturing synchronized dual-camera video and audio to collect authentic guest–staff interactions. Speech is transcribed using Whisper, and emotion is extracted from the corresponding video segments via DeepFace, producing 582 real Greek guest–receptionist exchange examples. The resulting data are classified into eight Standard Operating Procedure (SOP) categories. To address data scarcity, we augment the corpus with 1269 synthetic dialogues generated by eight diverse LLMs through the OpenRouter API, yielding a total of 1851 dialogue records with explicit emotion-token annotation. We fine-tune Qwen3.5-35B-A3B using Low-Rank Adaptation (LoRA) through a two-stage process: supervised fine-tuning (SFT) on an 888-example conversation pool and Simple Preference Optimization (SimPO) on a 1899-pair preference pool, each split 80/10/10 into training, validation, and test. The resulting model is integrated into an interactive hotel-terminal system combining YOLO-based person detection, face-recognition-driven guest personalization, Kokoro neural text-to-speech (TTS), and a multi-service orchestration layer connected to the hotel Property Management System (PMS). Evaluation combines standard text metrics, emotion-aware scoring, and a large-model judge. The results indicate targeted improvements in the rule-based contextual emotion-policy match and staff-emotion policy compliance compared to the base model, while general response-quality gains remain more modest. In particular, the rule-based contextual policy-match score improves from 0.614 to 0.901, while forbidden staff-emotion outputs decrease from 0.142 to 0.018. The deployed pilot demonstrates the practical integration of a personalized, emotion-aware LLM assistant in an interactive hotel-terminal setting; end-to-end latency and fully hotel-side edge deployment were not evaluated and are left for future work. HOSPIT-LLM provides a reproducible framework for multimodal dataset creation, preference-based fine-tuning, and deployment of human-centered AI systems. A mobile robotic embodiment is planned as future work.