Skip to content
Book Open access

Toward Codec-Native Interfaces for Wearable Video-Language Systems

Oct 2026 · Proceedings of the 4th ACM Workshop on Mobile Immersive Computing, Networking, and Systems · pp. 49-55 · 0 citations · 9 references

TL;DR

EventRetina is presented, a preliminary codec-native split-stream prototype that transmits low-frequency I-frame anchors and saliency-selected P-frame event deltas and improves the bandwidth-accuracy tradeoff over I-only streaming and random or uniform P-frame selection.

Abstract

Continuous video-language reasoning on wearable devices requires the edge to preserve useful visual evidence before the user query is known. Current pipelines often transmit sampled RGB frames, while standard codecs already organize video into I-frame appearance anchors and P-frame temporal updates. This paper explores whether such codec structure can serve as a lightweight edge-server interface for wearable Video-LM systems. We present EventRetina, a preliminary codec-native split-stream prototype that transmits low-frequency I-frame anchors and saliency-selected P-frame event deltas. The edge selects informative P-frames using codec-side motion and residual statistics under a bandwidth budget, while the server maintains the received anchors and deltas as visual evidence for later question answering. On the Perception Test dataset, EventRetina improves the bandwidth-accuracy tradeoff over I-only streaming and random or uniform P-frame selection. At a 20% P-frame budget, it reaches 46.3% accuracy while transmitting 0.80 Mbps. Under matched temporal evidence, it also reduces server-side visual processing cost compared with repeatedly encoding selected RGB frames. These results suggest that codec-native split-streaming is a promising design point for bandwidth-limited wearable Video-LM reasoning.

Read PDF

Similar papers

Preprint Oct 2026

EARS: Evidence-Aware Retrieval via Sound and Motion for Budgeted Video Reasoning

Uniform frame sampling can miss brief evidence in sparse-event videos when the VLM sees only K frames. We propose EARS (Evidence-Aware Retrieval via Sound and motion), a training-free, query-free front-end that ranks frames using optical-flow motion and audio energy, then applies temporal non-maximum suppression for di...

Shih-Yao Lin · 1 citation
Preprint Oct 2026

Event-guided Neural Video Compression

An Event-guided Neural Video Codec (ENVC) is proposed that uses events shared by the encoder and decoder to improve RGB compression efficiency and demonstrates the potential of events as a complementary modality for reducing the RGB coding rate.

Ji-Yun Kong, Jungwoo Kim, Enes Eray Demirtas et al. · 0 citations
Preprint Oct 2026

Video Encoders Built on Image Representations

The design of a video encoder determines when frames begin to interact and which frame-specific visual evidence remains accessible to the language model. Native video pathways couple neighboring frames during visual encoding, whereas image pathways preserve independently computed frame representations but incur a much...

Jusheng Zhang, Wen-Hao Wang, Long-Qi Cai et al. · 5 citations
Preprint Nov 2026

Transfer Once, Ask Locally: Compact Audio-Visual Memory for On-Device Question Answering

Audio-visual question answering on lightweight user devices requires balancing speed, memory usage, privacy, and communication. One approach is to run audio-visual encoders on a server and a lightweight language decoder on the user device, keeping questions local while offloading media encoding. However, transferring a...

Qi-Tong Wang, Ji-Hui Jin, Divya Kothandaraman et al. · 0 citations
Sep 2026

TT-VidT: Decoupling the Temporal Axis for Efficient Motion-Centric Video Pretraining

TT-VidT leads Jester, Something-Something V2, ARID, and Diving48 fine-tuning simultaneously, improving over the strongest non-TT row by 54% ~ 121%, while using 48% fewer encoder FLOPs than DisMo and 55% fewer than VideoMAE or V-JEPA2.

Shih-Ying Yeh, Daniel Z. Kaplan, Xue-Hai Wang et al. · 0 citations
Book Open access Oct 2026

ActiveSight: Enabling Continuous Wearable Perception through Algorithm-Hardware Co-Designed Multimodal Semantic Gating

Continuous wearable AI perception must capture transient, meaningful events throughout the day, yet always-on visual processing rapidly exhausts batteries of compact wearables. We present ActiveSight, an algorithm-hardware co-designed multimodal gating system that exploits the complementarity between audio and vision:...

Zhi-Hao Ren, Xiao-Yu Fang, Sohom Sen et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.