Action Descriptions as Semantic Priors for 3D Human Pose Estimation

Abstract

Human 3D pose estimation is an important problem in computer vision, having applications in a wide variety of fields, such as healthcare, sports science, human-computer interaction, entertainment and retail. However, when solely using 2D sequences of human joints as input data, this becomes an ill-posed problem, caused by the lack of a unique solution, since there are infinitely many 3D points which can be projected to the same 2D point. Current methods rely on temporal or anatomical priors for added information which can help the lifting process. In this work, we propose implementing an additional loss objective for the problem of 3D human pose estimation as a way of embedding semantic information into the lifting process by aligning projected skeleton features with CLIP text embeddings and show that such information can help improve skeleton lifting metrics, especially when applied on out-of-distribution data, without fine-tuning. Our method adds a small MLP projection head to an already existing 3D pose estimation network and a semantic loss between projected features and embeddings of action descriptions, while, at inference, it drops both components, making it adequate for application on novel data, without any prior knowledge of actions. Our code is publicly available at https://github.com/HRIA-HAR/SARPose.