Although VLM annotation required minutes per fly, the trained networks enabled pose estimation throughout more than 700,000 frames, demonstrating that VLM annotations can support fast animal pose estimation without manually labeled training images.
Abstract
Animal pose estimation typically requires manual annotation of body parts to train neural networks for each new recording system or environmental condition. Here we use a general-purpose vision-language model (VLM) to generate annotations in multi-animal recordings of fruit flies (Drosophila melanogaster). Initial annotations follow written anatomical descriptions and are reviewed by an independent VLM session using neighboring video frames. On 120 held-out fly images, AI annotations had a median distance of 2.28 pixels from human labels, with 84.0% agreement within eight pixels, averaged across landmark types. The comparison included 10 landmarks, with head and eyes analyzed separately because of systematic differences in placement. Review improved agreement primarily at the feet, with the largest remaining discrepancies at positions with limited visual evidence. Visual inspection revealed cases in which the AI annotations were more anatomically accurate than those of an experienced human annotator, particularly at the feet. A convolutional network trained using SLEAP on 300 AI-annotated fly images showed similar agreement with human labels and closely reproduced the AI annotations on held-out images, with a median distance of 1.56 pixels. We also applied the procedure to recordings from another laboratory, using new AI annotations to train a separate pose network. Although VLM annotation required minutes per fly, the trained networks enabled pose estimation throughout more than 700,000 frames. These results demonstrate that VLM annotations can support fast animal pose estimation without manually labeled training images.
Across benchmark videos of 10 mice, 10 fish, 32 beetles, 32 ants with one cricket, 60 flies, and 368 ants, AMADEUS outperformed state-of-the-art methods in identity tracking and keypoint localization.
Yusuke Notomi, Kentarou Matsumura, S. Dobata· bioRxiv· 0 citations
Automatically estimating animal poses from videos is important for studying animal behavior. Existing methods do not perform reliably, partly because they rely on high-quality training data, and existing such datasets are small. It is challenging and expensive to create such data, due to the large variations in animal...
Zhuo-Yang Pan, Bo-Xiao Pan, Guan-Dao Yang et al.· International Journal of Com...· 1 citation
Together, DeepLabSAM, MTMB dataset, and MouseLite provide a scalable framework for robust, cross-assay behavioural pose estimation of freely interacting mice, and substantially reduces misassigned keypoints under heavy occlusion.
Juan Cobos, Steeve Thirard, Marwen Belkaid et al.· bioRxiv· 0 citations
Animal focused computer vision datasets are of a growing interest in the fields of both Computer Vision and Animal behaviour. The ability to make sense of complex scenes via a variety of Computer Vision tasks has substantial implications for behavioural understanding and welfare. The perfection of these tasks will help...
Zane K. J. Hartley, Joseph Ho, Alan Chamberlain et al.· 0 citations
Human motion, environmental contacts, and interaction forces are governed by common physical laws, yet existing approaches typically separate visual pose reconstruction from contact and force estimation. This separation limits joint reasoning and can propagate errors between stages. We introduce PACT, an end-to-end mod...
Rikhat Akizhanov, Yang-Song Zhang, Nikolai Kaliazin et al.· 0 citations
Transferring visual representations pretrained on human videos to robot manipulation requires learning reliable correspondences between human and robot demonstrations. However, paired demonstrations can differ in execution rate and in the proportion of non-key frames that do not directly reflect task progress. Frames a...
Xi-Zhe Zhang, Jing-Feng Zhang, Zi-Run Zhou et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.