1255 Enhancing Surgical Computer Vision: A Real-Time Monitoring System for Model Performance and Data Quality

INTRODUCTION: As computer vision is increasingly applied to surgical clinical decision support, patient safety is threatened if models cannot be monitored for poor or false performance in real-time. We propose a monitoring system capable of identifying anomalies and errors in surgical computer vision applications without the need for ground truth labels. METHODS: Surgical videos were stored in a central data repository. A YOLOv8 model was used for object detection, trained on data labelled by manual annotators.The system employs a Vision Transformer to compare frames based on their similarity and flag anomalous frames. A similar process per class on bounding box crops identifies abnormal predictions. Surgery-specific heuristics detect logic-breaking predictions, such as multiple tool occurrences. Results are consolidated into a user-friendly report and automatically provided via electronic notifications. RESULTS: 829 frames were used for model development. Over the test set, 110 frames were flagged - 13% of total frames. Of those flagged, 88% required revision. Notably, the system accurately identifies frame outliers, such as blue screens simulating sleep screens. Engineers also receive statistics on monitoring results. CONCLUSIONS: Real-time, comprehensive assessment of model performance and data quality is critical to adoption of AI in surgery. This model monitoring system allows continuous improvements and risk mitigation in AI-surgeon collaboration. Further research is warranted to broaden its applicability and enhance surgical heuristics.