Skip to content
Book Open access

Multi-stream Semantic Understanding for Human-Object Interaction Detection

Nov 2026 · Proceedings of the 34th ACM International Conference on Multimedia · 0 citations · 44 references

Abstract

Human-Object Interaction (HOI) detection, a fundamental problem in visual scene understanding, relies heavily on contextual semantics to disambiguate complex interactions. Although Transformer-based methods have achieved notable progress, they largely remain instance-centric and insufficiently exploit the broader contextual cues that are crucial for human-like reasoning. To address this limitation, we propose MSSU (Multi-stream Semantic Understanding), a novel multi-stream HOI detection framework. MSSU uniquely integrates four complementary streams to capture diverse semantic cues, within which one stream leverages Stable Diffusion to directly inject human-like shape priors into the HOI detector. Furthermore, to effectively synthesize these streams, we design two novel fusion strategies: the Visual Geometry Encoder (VGE) and the Orthogonal Decomposition Fusion (ODF). The former fuses geometric and visual features, thereby enhancing the basic visual representations with relation-aware knowledge. The latter orthogonally fuses textures and shape priors, thus reducing the redundancy between these distinct features. Empowered by this multi-stream design, MSSU achieves state-of-the-art performance on the challenging HICO-DET and V-COCO benchmarks. Extensive experiments demonstrate that MSSU effectively captures diverse contextual semantics, enabling more robust and human-aligned reasoning for HOI detection.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.