A study of deep learning approaches for modeling temporal complexity, spatial context and class imbalance in visual analysis

Abstract

Deep learning models deployed in real-world vision applications must remain reliable under three fundamental challenges: temporal complexity, spatial ambiguity, and class imbalance. This dissertation introduces a unified set of architectures and training strategies designed to address these challenges across long-term video object segmentation and high-stakes medical imaging tasks. First, to overcome memory saturation, drift, and error propagation in long video sequences, we develop MuSMem, a scalable multi-object video object segmentation framework that incorporates Dynamic Smart Memory, selective keyframe updates, and SAM+DINO-guided refinement. By storing only the most informative embeddings using an information-preserving gain metric, MuSMem breaks the linear memory-growth limitation of AOT/DeAOT and achieves state-of-the-art robustness on VOTSt, LVOS, and long-video benchmarks even under occlusion, topology changes, and multi-object interactions. Second, to address appearance ambiguity in medical imaging, we introduce the Dual- Context Framework and Context-Guided Curriculum Learning for pancreatic lesion detection on the 10,000-scan PanTS dataset. The Dual-Context architecture jointly learns 27 abdominal organs and lesions to enforce anatomical priors, while the clinically derived curriculum schedules training from easy to hard cases using radiologist-defined difficulty metrics. This combined approach improves F1 to 0.7524, surpassing an imbalance-aware nnUNet baseline by 8.3 points and reducing false positives by 82 percent. Third, to improve robustness under long-tailed and heterogeneous clinical distributions, we develop several ensemble-based strategies. For neonatal hypoxic-ischemic encephalopathy (HIE), we construct a five-model Swin-UNETR ensemble that leverages long-range transformer representations to capture diffuse, variable-size lesions, ranking among the top three methods in the BONBID-HIE challenge. For retinal multi-disease classification, we design a strategic model-selection algorithm that identifies the most diverse and complementary subset of experts from the MExNet ensemble, improving multi-label disease detection and generalization to unseen populations without retraining. Finally, for facial age estimation--an extremely imbalanced biometric task--we combine diffusionbased synthetic data generation for minority age groups with an ensemble of deep regression models, substantially improving accuracy for underrepresented populations and stabilizing performance across demographic variations. Collectively, these contributions provide a coherent and principled framework that advances the scalability, reliability, and generalization of deep learning systems for both natural video analysis and clinical medical imaging. The resulting methods demonstrate that integrating efficient temporal modeling, explicit spatial structure, and imbalance-aware learning is essential for building trustworthy and deployable visual perception systems.