Domain-specific ControlNet training for neonatal pose-conditioned image generation

Abstract

Several automatic approaches have been proposed to analyse spontaneous movements in infants, but translation to clinical practice remains limited by the scarcity of neonatal data, especially in preterm populations. Generative models can help expand existing datasets, yet current solutions—such as generative adversarial networks or diffusion models—often fail to preserve anatomical and pose consistency. This study presents a ControlNet model specifically trained to drive the diffusion process for the generation of neonatal pose-conditioned images. Proprietary and publicly available infant videos were collected. By means of ViTPose and BLIP-2 we extracted frame-by-frame pose and captions, respectively, leading to 2141 pose-caption-image triplets. ControlNet was trained using Stable Diffusion v2.1 as pre-trained core model to generate pose-conditioned images. Eight training configurations were tested, varying in resolution, batch size, and gradient accumulation. Performance was evaluated against adult-trained ControlNet checkpoints using structural similarity index (SSIM), peak signal-to-noise ratio (PSNR), and learned perceptual image patch similarity (LPIPS). The best configuration outperformed the adult-trained model (SSIM: 0.453 vs. 0.086; PSNR: 12.81 vs. 6.35 dB; LPIPS: 0.626 vs. 0.926; all p < 0.001). High resolution and larger effective batch sizes improved convergence and visual fidelity. Downstream fine-tuning of the ViTPose model with the generated dataset showed significant pose detection improvements in unseen external infant videos ( p < 0.05). Domain-specific training markedly enhances neonatal image generation achieving superior structural and perceptual fidelity, with promising pose-related coherence. Their use may be crucial in synthetic data generation task for clinical motor assessments and rehabilitation.