Skip to content
Open access

A promptable foundation model enables automated multi-task dataset construction and real-time pose estimation in mice

Oct 2026 · bioRxiv · 0 citations · 41 references
Biology

TL;DR

Together, DeepLabSAM, MTMB dataset, and MouseLite provide a scalable framework for robust, cross-assay behavioural pose estimation of freely interacting mice, and substantially reduces misassigned keypoints under heavy occlusion.

Abstract

Behavioural experiments often hinge on moments when animals crowd together. In these situations, existing pose-estimation methods usually fail because detected crops include neighbouring animals, and their body parts are easily confused between individuals. We solved it with DeepLabSAM, a pipeline that segments and tracks every animal from a single text prompt using the SAM 3 foundation model, followed by a mask-gated pretrained pose estimation network from DeepLabCut. Across recordings of multiple interacting mice, the proposed method substantially reduces misassigned keypoints under heavy occlusion. Furthermore, we used DeepLabSAM to automatically construct the Multi-Task Mouse Behaviour (MTMB) dataset, comprising 14,300 frames spanning nine behavioural assays, with each instance annotated for bounding box, mask, identity, and pose. SuperAnimal pretrained pose models transferred unpredictably across this dataset. Conversely, MouseLite, our RF-DETR pose-estimation model trained on the MTMB dataset, matched their in-distribution accuracy, generalized better to unseen assays, and achieved real-time (faster-than-acquisition) inference. Combined, DeepLabSAM, MTMB dataset, and MouseLite provide a scalable framework for robust, cross-assay behavioural pose estimation of freely interacting mice.

Read PDF

Similar papers

Sep 2026

Beyond Known Objects: Prompting 6D Pose Estimator to Learn Novel Objects Continually

Robust 6D object pose estimation is essential for enabling intelligent systems to interact with the physical world. While existing methods have achieved notable progress, they remain constrained by two major limitations: poor scalability when new objects are introduced and severe performance degradation caused by catas...

Long Tian, Yang Liu, Jun-Lin Fang et al. · 0 citations
Open access Sep 2026

PosePAL: Efficient Animal Pose Labeling Using Point Trackers

Automatically estimating animal poses from videos is important for studying animal behavior. Existing methods do not perform reliably, partly because they rely on high-quality training data, and existing such datasets are small. It is challenging and expensive to create such data, due to the large variations in animal...

Zhuo-Yang Pan, Bo-Xiao Pan, Guan-Dao Yang et al. · 1 citation
Open access Oct 2026

Tri-channel synergistic model for 3D human pose estimation

3D human pose estimation from single-view images is a fundamental and challenging task in computer vision, the key of which lies in effectively capturing the complex spatial dependencies between human joint points. Existing methods usually rely on a single computational paradigm: Graph Convolutional Networks (GCN) exce...

Ying Gu, Cheng-Long Li, Xin-Yan Gao et al. · 0 citations
Book Oct 2026

ViPoser: Sparse-IMU Based Human Pose Estimation with Distilled Vision Foundation Priors

Human pose estimation using inertial measurement units (IMUs) has gained increasing attention due to its ability to enable camera-free motion capture for numerous human-centered applications. However, estimating full-body pose from only one to three daily carried commercial-off-the-shelf IMU devices (e.g., smartphones,...

Han-Yu Zeng, Hui Ji, Peng-Fei Zhou · 0 citations
Book Open access Nov 2026

Active Learning for Human Pose Estimation via Submodular Optimization on Latent Hyper-Ellipsoids

While deep neural networks have significantly advanced 2D Human Pose Estimation, their performance relies heavily on massive datasets with costly, dense skeletal annotations. In practical scenarios, while raw data is abundant, the high costs of manual annotation necessitate budget-constrained approaches. Active learnin...

Kadir Rabah, Julien Aligon, Xavier Baril et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.