Oct 2026· Proceedings of the 4th ACM Workshop on Mobile Immersive Computing, Networking, and Systems· pp. 49-55· 0 citations· 9 references
TL;DR
EventRetina is presented, a preliminary codec-native split-stream prototype that transmits low-frequency I-frame anchors and saliency-selected P-frame event deltas and improves the bandwidth-accuracy tradeoff over I-only streaming and random or uniform P-frame selection.
Abstract
Continuous video-language reasoning on wearable devices requires the edge to preserve useful visual evidence before the user query is known. Current pipelines often transmit sampled RGB frames, while standard codecs already organize video into I-frame appearance anchors and P-frame temporal updates. This paper explores whether such codec structure can serve as a lightweight edge-server interface for wearable Video-LM systems. We present EventRetina, a preliminary codec-native split-stream prototype that transmits low-frequency I-frame anchors and saliency-selected P-frame event deltas. The edge selects informative P-frames using codec-side motion and residual statistics under a bandwidth budget, while the server maintains the received anchors and deltas as visual evidence for later question answering. On the Perception Test dataset, EventRetina improves the bandwidth-accuracy tradeoff over I-only streaming and random or uniform P-frame selection. At a 20% P-frame budget, it reaches 46.3% accuracy while transmitting 0.80 Mbps. Under matched temporal evidence, it also reduces server-side visual processing cost compared with repeatedly encoding selected RGB frames. These results suggest that codec-native split-streaming is a promising design point for bandwidth-limited wearable Video-LM reasoning.
Uniform frame sampling can miss brief evidence in sparse-event videos when the VLM sees only K frames. We propose EARS (Evidence-Aware Retrieval via Sound and motion), a training-free, query-free front-end that ranks frames using optical-flow motion and audio energy, then applies temporal non-maximum suppression for di...
An Event-guided Neural Video Codec (ENVC) is proposed that uses events shared by the encoder and decoder to improve RGB compression efficiency and demonstrates the potential of events as a complementary modality for reducing the RGB coding rate.
Ji-Yun Kong, Jungwoo Kim, Enes Eray Demirtas et al.· 0 citations
The design of a video encoder determines when frames begin to interact and which frame-specific visual evidence remains accessible to the language model. Native video pathways couple neighboring frames during visual encoding, whereas image pathways preserve independently computed frame representations but incur a much...
Jusheng Zhang, Wen-Hao Wang, Long-Qi Cai et al.· 5 citations
Audio-visual question answering on lightweight user devices requires balancing speed, memory usage, privacy, and communication. One approach is to run audio-visual encoders on a server and a lightweight language decoder on the user device, keeping questions local while offloading media encoding. However, transferring a...
Qi-Tong Wang, Ji-Hui Jin, Divya Kothandaraman et al.· 0 citations
TT-VidT leads Jester, Something-Something V2, ARID, and Diving48 fine-tuning simultaneously, improving over the strongest non-TT row by 54% ~ 121%, while using 48% fewer encoder FLOPs than DisMo and 55% fewer than VideoMAE or V-JEPA2.
Shih-Ying Yeh, Daniel Z. Kaplan, Xue-Hai Wang et al.· arXiv.org· 0 citations
Continuous wearable AI perception must capture transient, meaningful events throughout the day, yet always-on visual processing rapidly exhausts batteries of compact wearables. We present ActiveSight, an algorithm-hardware co-designed multimodal gating system that exploits the complementarity between audio and vision:...
Zhi-Hao Ren, Xiao-Yu Fang, Sohom Sen et al.· Proceedings of the 32nd Annu...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.