Can Foundation Models See From Above? A Zero-Shot Evaluation of SAM 3 on Aerial Drone Imagery

Abstract

Drones increasingly depend on real-time object detection for surveillance, search-and-rescue, and infrastructure inspection, yet current onboard pipelines rely on supervised models such as You Only Look Once (YOLO) that must be retrained whenever the target class set changes. Open-vocabulary foundation models offer a possible escape from this retraining loop, but their effectiveness on the small objects, top-down viewpoints, and cluttered scenes typical of aerial imagery is unknown. I present, to my knowledge, the first quantitative evaluation of Segment Anything Model 3 (SAM 3) on drone footage, applying the model zero-shot to the VisDrone benchmark and the Semantic Drone Dataset using only plain-text class prompts. SAM 3 reaches a per-class mean Intersection-over-Union (IoU) of 0.241 on VisDrone (10 classes) and 0.282 on the Semantic Drone Dataset (22 classes), with strong results on large vehicles (car: 0.872 on Semantic Drone, 0.433 on VisDrone) and near-zero results on small or ambiguous classes (bicycle: 0.096; obstacle: 0.009). Recall consistently exceeds precision by roughly a factor of two, indicating systematic over-segmentation. The findings suggest SAM 3 is a useful starting point for flexible aerial perception, but is not yet a drop-in replacement for trained drone detectors, and they align with concurrent observations on other open-vocabulary detectors evaluated on aerial data.