An exciting and challenging research topic I am working on is content understanding in videos from surgical procedures and interaction with medical videos. These days many surgeons record videos during endoscopic, laparoscopy, or microscopic surgeries and archive them for later use, since these videos contain highly important information that can be used later for:
- teaching and training
- revisiting specific moments of the surgery for retrospective analysis
- explaining details to patients
- discussing details among surgeons
- reviewing technical skills
While this invaluable video data enables completely new use scenarios it also bears new challenges, such as managing the storage of a medical video archive – that increases on a daily basis – as well as automatic content analysis and understanding (e.g., phase recognition, instrument detection, surgical action classification, etc.) in order to make it efficiently accessible. These days deep learning methods are used to automatically detect and index relevant content in such videos.
Over the years we have investigated many different problems in this challenging domain. In the following you can find an overview of a few of our works.
Relevance-Based Compression of Cataract Videos
Problem/Idea: Only the relevant content of medical videos needs to be stored with highes quality. In this work we automatically detect relevant segments and regions-of-interest by using CNNs and Region-Based CNNs (Mask R-CNN) and use higher quantization for the H.265/HEVC encoding of other irrelevant content (or even remove it completely).
Paper: https://doi.org/10.1145/3394171.3413658
Pupil Reaction Detection
Problem/Idea: In cataract surgery some patients show pupil reactions during the surgery, which occur with contractions of the eye chamber and hence can be very dangerous. It is a topic of research why these reactions occur. Therefore, it is important to track them among many operations and patients. In this work, we automatically segment the pupil and iris (i.e., cornea) and analyze cataract surgery videos for such pupil reactions.
Paper: https://doi.org/10.1371/journal.pone.0258390
Endometriosis Segmentation Tool
Problem/Idea: Endometriosis is a pain- and often harmful disease that affects the female reproductive system. In this work we developed an automatic detection of tissue affected by endometriosis. The tool comes with a pre-trained model learned from many manual annotations and can be used for testing purpose.
Paper: https://doi.org/10.1007/s11042-021-11730-1
https://github.com/amplejoe/EndometriosisSegmentationTool
Medical Video Exploration
Problem/Idea: We need video interaction and retrieval tools to effectively search in large video archives of medical videos. The images show the architecture of two different medical video exploration systems, while the video shows one system for exploration of laparoscopic videos, with features like content-based search, visual exploration, and similarity search.
Papers:
Object Segmentation in Medical Videos
Problem/Idea: In order to understand the content of medical videos, it is important to automatically detect relevant objects, such as instruments, tools, anatomical structures. We have investigated several different object segmentation/recognition methods for this.
Papers:
- https://doi.org/10.1007/978-3-030-92238-2_33
- https://doi.org/10.1109/CBMS49503.2020.00112
- https://doi.org/10.1016/j.media.2020.101920
