DART shows that freely available geometric pseudo-labels can strengthen foundation model pretraining without extra labels or added inference cost, pointing toward stronger backbones for surgery.
Abstract
Vision foundation models (VFMs) are valuable in data-scarce domains such as surgery, where a single pretrained backbone can provide rich representations for many downstream tasks. Yet the dominant self-supervised pretraining paradigm uses only RGB images, leaving readily available complementary signals, such as depth maps, unused. This is a particular missed opportunity in surgery, where natural-image VFMs transfer poorly while the scene geometry is rich and informative. With strong off-the-shelf models now able to produce pseudo-labeled dense depth for any image corpus, we hypothesize that such signals can be folded into pretraining to learn better representations. We present DART, an RGB-D pretraining recipe that builds on DINOv2 with a simple modification: a pixel-space depth reconstruction objective applied to masked iBOT patches, supervised by pseudo-labeled depth. Depth is used only during pretraining, so fine-tuning and inference remain RGB-only. We find that this pixel-level reconstruction head improves representation quality rather than disrupting it. We further show that depth, which encodes scene geometry, is more effective as a target than alternative dense signals such as Canny edges, confirming that the gains stem from depth rather than added supervision alone. Across eight surgical benchmarks spanning segmentation, depth estimation, and image-level recognition, DART outperforms both natural-image and in-domain baselines, including a vanilla DINOv2 trained on identical data, improving dense prediction while also strengthening image-level understanding. More broadly, DART shows that freely available geometric pseudo-labels can strengthen foundation model pretraining without extra labels or added inference cost, pointing toward stronger backbones for surgery.
Vision foundation models are the backbone of modern surgical image analysis, yet most surgical systems still rely on encoders pretrained on natural images or on specialist models trained from scratch on comparatively small datasets. Pretraining encoders on large amounts of surgery-specific data has the potential to gen...
Kun Yuan, Hao Ding, Yi-Qing Shen et al.· 0 citations
Through the first systematic study of TTA for dense depth regression (eight baselines), it is shown that existing TTA methods consistently fail in this setting: entropy-based methods degrade regardless of the number of adapted parameters, and temporal-photometric adaptation diverges on non-rigid tissue.
Sheng-Yao Wang, Jun-Pei Xue, Xin Zhong et al.· IEEE Transactions on Medical...· 1 citation
The core objective of deformable image registration (DIR), estimating a displacement vector field that aligns two images, is fundamentally the same across imaging modalities and anatomical regions. Although this shared objective should imply the use of foundation models, most deep learning approaches for DIR operate wi...
Wenke Karbole, Vasiliki Sideri-Lampretsa, Hua-Qi Qiu et al.· IEEE Transactions on Medical...· 0 citations
Analysis of routing utilization, top-k selection, trainable parameters, and inference latency demonstrate that IC-DVLoRA provides an effective and efficient adaptation strategy for endoscopic depth foundation models.
Ze-Yun Liu, Ke Niu, Xue Feng et al.· Proceedings of the 35th ACM...· 0 citations
The key insight is that DA3's cross-view features, though trained only for depth consistency, implicitly encode motion-discriminative signals when combined with best-match feature search across frames, which could enable much denser temporal sampling that prior methods cannot support.
Xin-Hao Xiang, Wei-Yang Li, Zhi-Jie Zheng et al.· 0 citations
It is found that MultiMAE is a surprisingly strong out-of-the-box baseline, transferring effectively even to modalities it has never seen, while targeted modifications yield incremental but consistent improvements on multi-sensor transfer and overall robustness.
Fabio Montello, Ronja Güldenring, Lucas M. Sandby et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.