Automated Microscopy Video Object Detection of Motile Cells With Static-Dynamic Transformers

Abstract

Automated video object detection (VOD) of motile cells under microscopy is important for characterizing their motility and morphology properties. It poses significant challenges beyond object detection in static images due to motion blur, out-of-focus frames, and frequent pose variations. Compared with VOD in generic scenes, the lower contrast and smaller color space of microscopy imaging further introduce feature overlap between the foreground and background objects (e.g., impurity cells and contaminants). Transformer-based methods have achieved success in the VOD of generic scenes by utilizing object queries to model the inner-frame and inter-frame objects. However, the appearance overlap problem in microscopy video frames significantly compromises the inter-frame query aggregation by introducing background features into the object query. To overcome this challenge, we propose a static-dynamic query-based VOD network that treats object queries of the current video frame and reference video frames differently. The framework operates in two stages: it first generates high-quality object queries from reference frames using a static Transformer decoder pre-trained on a still image dataset, followed by training a dynamic Transformer decoder on a per-frame annotated dataset to handle object queries in the current frame. Experiments on a clinically collected biopsied sperm dataset validated the effectiveness of the proposed method. Note to Practitioners—Video object detection (VOD) of motile cells has various automation applications in life sciences, such as cell migration tracking and motility & morphology characterization. State-of-the-art methods employ object queries generated by Transformer decoders to model objects in different video frames. However, the appearance overlap problem in microscopy imaging can introduce background features into the object queries, leading to inaccurate detections. To address this issue, this paper proposes a VOD framework based on static-dynamic Transformers that separately model the object queries of the current frame and reference frames. To support the feature aggregation with the current queries, a static Transformer decoder is pre-trained on a still image dataset, allowing it to generate high-quality reference queries enriched with foreground features. A reference query encoder is designed to enhance the reference queries with instance-aware information by rescaling the spatial attention among them. Additionally, a contrastive query learning strategy is proposed to mitigate the duplicate matching problem during inference by encouraging query embeddings of the ground truth queries to be more distinct from other queries in the feature space. The proposed method can be effectively expanded into other medical VOD scenarios where the foreground targets share little feature difference with the background elements.