A Multimodal Weighted-Fusion Framework for Real-Time Suspicious Activity Detection Using EfficientNet-B0, MediaPipe Pose Landmarks, and XGBoost

Abstract

Detecting suspicious human activity in indoor surveillance environments is a safety-critical task where reliance on a single visual modality consistently leads to performance degradation under real-world conditions. Appearance-based convolutional models struggle with lighting variation and cluttered backgrounds, while landmark-based structural models break down whenever body occlusion prevents reliable keypoint extraction. This paper describes a dual-branch system that processes each camera frame through two independent pathways simultaneously: a two-stage fine-tuned EfficientNet-B0 network that operates on raw pixel content, and a MediaPipe Pose extractor whose 66-dimensional body landmark vector is passed to a calibrated XGBoost ensemble. Frame-level probabilities from both pathways are merged via a weighted linear rule, with the scalar mixing coefficient determined through an exhaustive 101-step grid search on the validation partition. Experiments on a self-collected dataset of 1,140 labelled images from four participants show individual accuracies of 95.4% and 94.0% for the appearance and pose branches respectively, rising to 96.8% (95% bootstrap CI: 93.5%–99.1%) with an <inline-formula> <tex-math notation="LaTeX">$F_{1}$ </tex-math></inline-formula>-score of 0.961 after fusion on a 217-sample held-out set, with a directional 75% reduction in false positives (McNemar <inline-formula> <tex-math notation="LaTeX">$p = 0.0703$ </tex-math></inline-formula>; preliminary finding pending larger-scale validation). When pose extraction fails, the system falls back to appearance-only prediction without any interruption to the inference loop. Five-fold cross-validation confirms that the pose classifier generalises stably, and McNemar’s test quantifies the directional advantage of fusion, though this preliminary advantage does not yet reach statistical significance at the 0.05 threshold (<inline-formula> <tex-math notation="LaTeX">$p = 0.0703$ </tex-math></inline-formula>, <inline-formula> <tex-math notation="LaTeX">$n = 217$ </tex-math></inline-formula>) and warrants validation on a larger dataset. Corrected end-to-end model inference throughput reaches 38.9 frames per second as a conservative lower bound, exceeding the 25 FPS real-time threshold by a factor of 1.6; actual GPU throughput is higher as the MediaPipe component was timed on CPU hardware. The working system is packaged as a Streamlit application that performs temporal smoothing, sends automated email alerts on confirmed detections, and saves incident clips to disk. The extracted pose landmark dataset is publicly available at <uri>https://doi.org/10.5281/zenodo.19914642</uri>.