Medical Video Understanding

An exciting and challenging research topic I am working on is content understanding in videos from surgical procedures and interaction with medical videos. These days many surgeons record videos during endoscopic, laparoscopy, or microscopic surgeries and archive them for later use, since these videos contain highly important information that can be used later for:

  • teaching and training
  • revisiting specific moments of the surgery for retrospective analysis
  • explaining details to patients
  • discussing details among surgeons
  • reviewing technical skills

While this invaluable video data enables completely new use scenarios it also bears new challenges, such as managing the storage of a medical video archive – that increases on a daily basis – as well as automatic content analysis and understanding (e.g., phase recognition, instrument detection, surgical action classification, etc.) in order to make it efficiently accessible. These days deep learning methods are used to automatically detect and index relevant content in such videos.

Over the years we have investigated many different problems in this challenging domain. In the following you can find an overview of a few of our works.

Relevance-Based Compression of Cataract Videos

Problem/Idea: Only the relevant content of medical videos needs to be stored with highes quality. In this work we automatically detect relevant segments and regions-of-interest by using CNNs and Region-Based CNNs (Mask R-CNN) and use higher quantization for the H.265/HEVC encoding of other irrelevant content (or even remove it completely).


Pupil Reaction Detection

Problem/Idea: In cataract surgery some patients show pupil reactions during the surgery, which occur with contractions of the eye chamber and hence can be very dangerous. It is a topic of research why these reactions occur. Therefore, it is important to track them among many operations and patients. In this work, we automatically segment the pupil and iris (i.e., cornea) and analyze cataract surgery videos for such pupil reactions.


Endometriosis Segmentation Tool

Problem/Idea: Endometriosis is a pain- and often harmful disease that affects the female reproductive system. In this work we developed an automatic detection of tissue affected by endometriosis. The tool comes with a pre-trained model learned from many manual annotations and can be used for testing purpose.


Medical Video Exploration

Problem/Idea: We need video interaction and retrieval tools to effectively search in large video archives of medical videos. The images show the architecture of two different medical video exploration systems, while the video shows one system for exploration of laparoscopic videos, with features like content-based search, visual exploration, and similarity search.


Object Segmentation in Medical Videos

Problem/Idea: In order to understand the content of medical videos, it is important to automatically detect relevant objects, such as instruments, tools, anatomical structures. We have investigated several different object segmentation/recognition methods for this.