arXiv preprint ·
SurGPT: A Multi-Task Foundation Model for Robotic Surgical Skills Assessment and Gesture Recognition
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
The assessment of surgical skills in robotic surgery training has traditionally relied on subjective evaluations by human experts, creating bottlenecks in surgical education and limiting scalability. We present a novel foundation model that automatically analyzes robotic surgical videos to provide comprehensive skill assessment and gesture understanding. Combining a visual encoder with a transformer architecture, the model performs three key tasks simultaneously: 1) surgical task classification (knot tying, needle passing, suturing), 2) skill level assessment on a standardized scale, and 3) natural language generation of surgical gesture descriptions. The model processes video sequences from the da Vinci surgical robot and generates human-readable text descriptions of surgical actions, similar to how language models generate text from prompts, using a multi-task learning framework with adaptive uncertainty weighting to optimize performance across all objectives. Our transformer-based architecture incorporates positional encoding for temporal understanding and attention mechanisms for spatiotemporal feature integration. Evaluated on the da Vinci Skills Simulator dataset, the proposed framework demonstrates state-of-the-art performance in surgical task recognition and provides interpretable skill assessments that correlate with expert evaluations, with the model's ability to generate natural language descriptions of surgical gestures opening new possibilities for automated surgical education, real-time feedback systems, and objective skill assessment in surgical training and certification programs.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Robotics
Dream What I Can: Sparse Visual Planning for Long-Horizon Robot Manipulation
Vector-Quantized World Model: Latent Feedback Motion Planning for Robot Manipulation
Scalable, Intuitive, Mixed Reality-Based Semi-Autonomous Robotic Telemanipulation for the Execution of a Fine-Grained Bimanual Task
Humanoid SRL: Augmenting Humanoids with Supernumerary Limbs for Coordinated Loco-Manipulation
Finite-Time Capture by a Dubins Pursuer Against Unknown Evader Strategies
Get the few papers worth your time in Robotics every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.