Medical Video Understanding

An exciting and challenging research topic I am working on is content understanding in videos from surgical procedures and interaction with medical videos. The rational behind is the fact that these days many surgeons record videos during endoscopic, laparoscopy, or microscopic surgeries and archive them for later use, since these videos contain highly important information that can be used later for:

  • teaching and training
  • revisiting specific moments of the surgery for retrospective analysis
  • explaining details to patients
  • discussing details among surgeons
  • reviewing technical skills

While this invaluable video data enables completely new use scenarios it also bears new challenges, such as managing the storage of a medical video archive – that increases on a daily basis – as well as automatic content analysis and understanding (e.g., phase recognition, instrument detection, surgical action classification, etc.) in order to make it efficiently accessible. These days deep learning methods are used to automatically detect and index relevant content in such videos. A broad overview of this special field of medical multimedia (including its challenges and opportunities, as well as different aspects and specialities) can be found in the slides of our tutorial, presented recently at the ACM Multimedia 2019 conference in Nice, France (slides on the right).