Skip to content

Static and Dynamic Prototype Enhanced Multi-Modal Models for Pedestrian Intention Understanding

2026 · IEEE transactions on multimedia · Vol 28, pp. 17407-17420 · 0 citations · 66 references

Abstract

Pedestrian intention prediction plays a vital role in improving decision-making in autonomous driving systems enhancing road safety, especially in intricate traffic environments. However, this task faces significant challenges due to intent ambiguity, behavioral diversity, and temporal variability in the data.Previous studies have primarily focused on multi-modal information fusion to mitigate intent ambiguity, but do not effectively explore the behavioral prior knowledge in training datasets or leverage temporal features within frames, limiting their ability to accurately capture pedestrian intent in dynamic and complex environments. Inspired by the strengths of prototype learning in effectively aggregating information from diverse sources, we propose Static and Dynamic Prototype enhanced Multi-modal Model (SDP-MM) for robustly understanding pedestrian intention.Specifically, our framework is built on a vision-language model (VLM) to effectively extract information from multi-modal signals. Then, to enhance behavior identification, we introduce a Static Prior Prototype (SPP) module that aggregates behavioral patterns from the training set, providing strong prior knowledge for identification. Additionally, to better model temporal dependencies in the data, we design a Dynamic Memory Prototype (DMP) module, which captures both long-term and short-term information across frames. These two modules complement each other, strengthening the framework's ability to handle dynamic and uncertain conditions in complex environments. Extensive experiments on two benchmarks highlight the superiority of our method over existing state-of-the-art approaches. Notably, it achieves over a 10% improvement in accuracy on the JAADbeh.

View source

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.