Title: Estimating future forceps movement using deep learning for robotic camera control in laparoscopic surgery

Authors: Yamato Umetani; Masahiko Minamoto; Shigeki Hori; Tetsuro Miyazaki; Kenji Kawashima

Addresses: Faculty of Engineering, Chiba University, Japan ' Monozukuri Engineering Department, Tokyo Metropolitan College of Industrial Technology, Japan ' Monozukuri Engineering Department, Tokyo Metropolitan College of Industrial Technology, Japan ' Department of Information Physics and Computing, School of Information Science and Technology, The University of Tokyo, Japan ' Department of Information Physics and Computing, School of Information Science and Technology, The University of Tokyo, Japan

Abstract: In laparoscopic surgery, an assistant surgeon must hold the laparoscope. Autonomous control of the holder helps the surgeon focus on the surgery. Estimating the future movement of the forceps can improve the control performance of a holder robot. We have previously proposed a method for estimating the position of the forceps 0.1 seconds ahead in a 2D image by using deep learning, based on segmented forceps in the camera image. In this study, we extend the prediction time to 0.1-3 seconds ahead, and investigate the accuracy of the estimation when the number of past position inputs to the convolutional neural network changes. We confirm that the forceps position in a 2D image 0.8 seconds ahead can be estimated online using only the past three positions in the suturing task within an error of 30 pixels, which is acceptable for laparoscope holder control.

Keywords: laparoscopic surgery; camera holder; motion estimation; semantic segmentation; deep learning; neural network; forceps; mean intersection over union; MIoU; pre-training; learning rate.

DOI: 10.1504/IJMA.2022.122323

International Journal of Mechatronics and Automation, 2022 Vol.9 No.2, pp.72 - 80

Received: 14 Sep 2021
Accepted: 25 Oct 2021
Published online: 19 Apr 2022 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article