Underwater sea turtle behavior recognition: a lightweight pose-to-action pipeline

Automated recognition of animal behaviors is an important computer vision task that improves ecological monitoring and behavioral analysis. Compared to generic human action recognition, these applications often suffer from severe constraints such as low onsite computational power, limited data availability for training learning-based models, and suboptimal image quality due to environmental conditions. For sea turtles, behavior in relation to fishing gear is particularly important for understanding and reducing the bycatch, or incidental take, and associated mortality. Monitoring such behavior underwater is challenging because viewing angles vary over time, and pose and motion trajectories are highly dependent on the camera angle. In this study, we address this problem with a compact pose-to-action pipeline that detects a small set of turtle morphological keypoints in each frame, and then classifies short sequences of keypoints into U-turn, reversal, or other routine behaviors. While we employ the YoloV8n pose model for keypoint detection, we use a shallow fully connected network for classifying the behavior types. Our two-stage training strategy allows us to train our pose estimation network with real data while optimizing the behavior recognition network with both real annotated clips and a large set of simulated trajectories including various camera geometries and motion parameters. We further reduced the computational requirements by finding a balance between the input frame rate and recognition accuracy. Our experimental results show that we can achieve 93.2% recognition accuracy with a minimum frame rate requirement of 2.86fps.