Skip to content
Review Open access

Learning fly pose from vision-language annotations

Oct 2026 · bioRxiv · 0 citations · 32 references
Biology

TL;DR

Although VLM annotation required minutes per fly, the trained networks enabled pose estimation throughout more than 700,000 frames, demonstrating that VLM annotations can support fast animal pose estimation without manually labeled training images.

Abstract

Animal pose estimation typically requires manual annotation of body parts to train neural networks for each new recording system or environmental condition. Here we use a general-purpose vision-language model (VLM) to generate annotations in multi-animal recordings of fruit flies (Drosophila melanogaster). Initial annotations follow written anatomical descriptions and are reviewed by an independent VLM session using neighboring video frames. On 120 held-out fly images, AI annotations had a median distance of 2.28 pixels from human labels, with 84.0% agreement within eight pixels, averaged across landmark types. The comparison included 10 landmarks, with head and eyes analyzed separately because of systematic differences in placement. Review improved agreement primarily at the feet, with the largest remaining discrepancies at positions with limited visual evidence. Visual inspection revealed cases in which the AI annotations were more anatomically accurate than those of an experienced human annotator, particularly at the feet. A convolutional network trained using SLEAP on 300 AI-annotated fly images showed similar agreement with human labels and closely reproduced the AI annotations on held-out images, with a median distance of 1.56 pixels. We also applied the procedure to recordings from another laboratory, using new AI annotations to train a separate pose network. Although VLM annotation required minutes per fly, the trained networks enabled pose estimation throughout more than 700,000 frames. These results demonstrate that VLM annotations can support fast animal pose estimation without manually labeled training images.

Read PDF

Similar papers

Open access Sep 2026

PosePAL: Efficient Animal Pose Labeling Using Point Trackers

Automatically estimating animal poses from videos is important for studying animal behavior. Existing methods do not perform reliably, partly because they rely on high-quality training data, and existing such datasets are small. It is challenging and expensive to create such data, due to the large variations in animal...

Zhuo-Yang Pan, Bo-Xiao Pan, Guan-Dao Yang et al. · 1 citation
Open access Oct 2026

A promptable foundation model enables automated multi-task dataset construction and real-time pose estimation in mice

Together, DeepLabSAM, MTMB dataset, and MouseLite provide a scalable framework for robust, cross-assay behavioural pose estimation of freely interacting mice, and substantially reduces misassigned keypoints under heavy occlusion.

Juan Cobos, Steeve Thirard, Marwen Belkaid et al. · 0 citations
Preprint Nov 2026

Cat Royale Comprehensive Dataset: A Dataset for Pose Detection, Segmentation, Animal Action Recognition and Scene Understanding

Animal focused computer vision datasets are of a growing interest in the fields of both Computer Vision and Animal behaviour. The ability to make sense of complex scenes via a variety of Computer Vision tasks has substantial implications for behavioural understanding and welfare. The perfection of these tasks will help...

Zane K. J. Hartley, Joseph Ho, Alan Chamberlain et al. · 0 citations
#artificial intelligence Preprint Sep 2026

PACT: End-to-End Learning of Human Pose, Contacts, and Forces from Video

Human motion, environmental contacts, and interaction forces are governed by common physical laws, yet existing approaches typically separate visual pose reconstruction from contact and force estimation. This separation limits joint reasoning and can propagate errors between stages. We introduce PACT, an end-to-end mod...

Rikhat Akizhanov, Yang-Song Zhang, Nikolai Kaliazin et al. · 0 citations
#machine learning Preprint Oct 2026

Frame-Level Temporal Alignment for Human-to-Robot Visual Adaptation

Transferring visual representations pretrained on human videos to robot manipulation requires learning reliable correspondences between human and robot demonstrations. However, paired demonstrations can differ in execution rate and in the proportion of non-key frames that do not directly reflect task progress. Frames a...

Xi-Zhe Zhang, Jing-Feng Zhang, Zi-Run Zhou et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.