基于计算机视觉的人体内腔三维重建技术综述

doi:10.19678/j.issn.1000-3428.0061710

摘要/Abstract

摘要： 医生通过内窥镜观察的人体内腔显示为二维图像，不能立体地展现内腔环境中病灶、血管及邻近组织的关系，而内腔三维重建及可视化技术能够清晰、全面地展现病灶及其他组织的三维形态，更好地辅助医生进行精准的手术判断。将人体内腔环境中的三维重建技术分为主动式测量方法与被动式测量方法，分类综述基于结构光、飞行时间、双目立体视觉、单目视觉的内腔三维重建技术及发展现状。针对同时定位与地图构建的内腔三维重建法，分析对比内腔环境下的特征点检测与匹配的发展、方法及特点，并对人体内腔三维重建的难点和未来发展趋势进行展望。

关键词: 内腔三维重建, 结构光, 飞行时间, 双目立体视觉, 单目视觉, 特征点检测, 特征点匹配

Abstract: The human lumen images observed by the doctor through the endoscope is shown in a two-dimensional style, and fail to present the three-dimensional relationships between the lesions, blood vessels and adjacent tissues in the human lumen.The three-dimensional human lumen reconstruction technology and visualization methods can clearly and comprehensively display the three-dimensional images of lesions and tissues, assisting doctors in precise operations.In this paper, the three-dimensional reconstruction technologies for human lumen are divided into the active measurement methods and passive measurement methods.On this basis, this paper summarizes the reconstruction technologies based on structured light, Time Of Flight(TOF), binocular stereo vision, and monocular vision in order, and discusses their development.Then focusing on the reconstruction technology using Simultaneous Localization and Mapping(SLAM), the paper analyzes and compares the development, methods and characteristics of the feature point detection and matching in human lumen.Finally, the difficulties and future development trends of the three-dimensional human lumen reconstruction technology are prospected.

Key words: 3D lumen reconstruction, structured light, Time Of Flight(TOF), binocular stereo vision, monocular vision, feature point detection, feature point matching

中图分类号:

TP391

吴海滨, 徐若彤, 王爱丽, 于晓洋, 岩堀祐之, 赵蓝飞, 刘赫. 基于计算机视觉的人体内腔三维重建技术综述[J]. 计算机工程, 2021, 47(10): 1-15.

WU Haibin, XU Ruotong, WANG Aili, YU Xiaoyang, IWAHORI Yuji, ZHAO Lanfei, LIU He. Overview of 3D Human Lumen Reconstruction Technology Based on Computer Vision[J]. Computer Engineering, 2021, 47(10): 1-15.

https://www.ecice06.com/CN/Y2021/V47/I10/1

图/表 15

20211015185751

20211015185755

20211015185759

20211015185803

20211015185806

20211015185810

20211015185813

20211015185816

20211015185820

20211015185823

20211015185827

20211015185831

20211015185835

20211015185842

20211015185848

参考文献

[1] ROSEN M, PONSKY J.Mininmally invasive surgery[J].Endoscopy, 2001, 33(4):358-364.
[2] 费保蔚.计算机辅助外科手术的方法和进展[J].国际生物医学工程杂志, 2015, 20(4):624-631. FEI B W.The method and development of compute-assisted surgery[J].International Journal of Biomedical Engineering, 2015, 20(4):624-631.(in Chinese)
[3] DISSANAYZKE G, DURRANT-WHYTE H, BAILEY T.A computationally efficient solution to the Simultaneous Localisation and Mapping(SLAM) building problem[C]//Proceedings of IEEE International Conference on Robotics & Automation.Washington D.C., USA:IEEE Press, 2000:1009-1014.
[4] DURRANT-WHYTE H, MAJUMDER S, THRUN S, et al.A Bayesian algorithm for simultaneous localisation and map building[J].Robotics Research, 2003, 264(3):49-60.
[5] DISSANAYZKE G, NEWMAN P, CLARY S, et al.A solution to the simultaneous localisation and mapping building problem[J].IEEE Transactions of Robotics and Automation, 2013, 17(3):229-241.
[6] 佟帅, 徐晓刚, 易成涛, 等.基于视觉的三维重建技术综述[J].计算机应用研究, 2011, 28(7):2411-2417. TONG S, XU X G, YI C T, et al.Overview on vision-based 3D reconstruction[J].Application Research of Computers, 2011, 28(7):2411-2417.(in Chinese)
[7] HANEISHI H, OGURA T, MIYAKE Y.Profilometry of a gastrointestinal surface by an endoscope with laser beam projection[J].Optics Letters, 1994, 19(9):601-603.
[8] 郑太雄, 黄帅, 李永福, 等.基于视觉的三维重建关键技术研究综述[J].自动化学报, 2020, 46(4):631-652. ZHENG T X, HUANG S, LI Y F, et al.Key techniques for vision based 3D reconstruction:a review[J].Acta Automatica Sinica, 2020, 46(4):631-652.(in Chinese)
[9] ARMBRUSTER K, SCHEFFLER M.Messendes 3D-endoskop[EB/OL].[2021-04-01].https://philpapers.org/rec/ARMMD.
[10] HAYASHIBE M, SUZUKI N, NAKAMURA Y.Laser-scan endoscope system for intraoperative geometry acquisition and surgical robot safety management[J].Medical Image Analysis, 2006, 10(4):509-519.
[11] SALVI J, PAGÈS J, BATLLE J.Pattern codification strategies in structured light systems[J].Pattern Recognition, 2004, 37(4):827-849.
[12] ALBITAR C, GRAEBLING P, DOIGNON C.Fast 3D vision with robust structured light coding[C]//Proceedings of IEEE SPIE'09.Washington D.C., USA:IEEE Press, 2009:125-136.
[13] MAURICE X, ALBITAR C, DOIGNON C, et al.A structured light-based laparoscope with real-time organs' surface reconstruction for minimally invasive surgery[C]//Proceedings of 2012 IEEE International Conference on Engineering in Medicine and Biology Society.Washington D.C., USA:IEEE Press, 2012:5769-5772.
[14] CLANCY N T, STOYANOV D, MAIER-HEIN L, et al.Spectrally encoded fiber-based structured lighting probe for intraoperative 3D imaging[J].Biomedical Optics Express, 2011, 2(11):3119-3128.
[15] REITER A, SIGARAS A, FOELER D, et al.Surgical structured light for 3D minimally invasive surgical imaging[C]//Proceedings of IEEE/RSJ International Conference on Intelligent Robots & Systems.Washington D.C., USA:IEEE Press, 2014:1282-1287.
[16] QIU L, REN H L.Endoscope navigation and 3D reconstruction of oral cavity by visual SLAM with mitigated data scarcity[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2018:1354-1476.
[17] HAMADON A B, DAUL C, SOUSSEN C, et al.A novel 3D surface construction approach:application to three-dimensional endoscopic data[C]//Proceedings of IEEE International Conference on Image Processing.Washington D.C., USA:IEEE Press, 2010:235-246.
[18] YANG S P, SEO Y H, HWANG K, et al.Switchable dot projection module for 3D flexible endoscopes[C]//Proceedings of 2019 IEEE International Conference on Optical MEMS and Nanophotonics.Washington D.C., USA:IEEE Press, 2019:346-358.
[19] FURUKAWA R, OKA S, KOTACHI T, et al.Fully auto-calibrated active-stereo-based 3D endoscopic system using correspondence estimation with graph convolutional network[C]//Proceedings of Annual International Conference of the IEEE Engineering in Medicine and Biology Society IEEE Engineering in Medicine and Biology Society Annual International Conference.Washington D.C., USA:IEEE Press, 2020:4357-4360.
[20] 李占利, 周康, 牟琦, 等.TOF相机实时高精度深度误差补偿方法[J].红外与激光工程, 2019, 48(12):263-272. LI Z L, ZHOU K, MU Q, et al.TOF camera real-time high precision depth error compensation method[J].Infrared and Laser Engineering, 2019, 48(12):263-272.(in Chinese)
[21] 魏加立, 曲慧东, 王永宪, 等.基于飞行时间法的3D相机研究综述[J].红外技术, 2021, 43(1):60-67. WEI J L, QU H D, WANG Y X, et al.Research review of 3D cameras based on time-of-flight method[J].Infrared Technology, 2021, 43(1):60-67.(in Chinese)
[22] TATAR F, MOLLINGER J R, DULK R, et al.Ultrasonic sensor system for measuring position and orientation of laproscopic instruments in minimal invasive surgery[C]//Proceedings of the 2nd Annual International IEEE-EMB Special Topic Conference on Micro Technologies in Medicine & Biology.Washington D.C., USA:IEEE Press, 2002:113-125.
[23] PENNE J, SCHALLER C, HORNEGGER J, et al.Robust real-time 3D respiratory motion detection using time-of-flight cameras[J].International Journal of Computer Assisted Radiology and Surgery, 2008, 3(5):427-431.
[24] PENNE J, SCHRAUDER T, SCHNEIDER A, et al.Time-of-flight 3D endoscopy[C]//Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention.London, UK:[s.n.], 2009:467-474.
[25] HAASE S, WASZA J, KILGUS T, et al.Laparoscopic instrument localization using a 3D time-of-flight/RGB endoscope[C]//Proceedings of 2013 IEEE Workshop on Applications of Computer Vision.Washington D.C., USA:IEEE Press, 2013:478-486.
[26] ZVOLSKY M, CSERKASZKY A, CUCCIATI G, et al.Monte-carlo simulations for an endoscopic time-of-flight PET detector[C]//Proceedings of 2014 IEEE Nuclear Science and Medical Imaging Conference.Washington D.C., USA:IEEE Press, 2014:253-268.
[27] ZORRAQUINO C, BUGALHO R, ROLO M, et al.Asymmetric data acquisition system for an endoscopic PET-US detector[J].IEEE Transactions on Nuclear Science, 2016, 63(1):213-221.
[28] BRANDT O, MUNWES Y.Commissioning and first image reconstruction with a new time-of-flight PET prototype[C]//Proceedings of 2018 IEEE Conference on Nuclear Science and Medical Imaging.Washington D.C., USA:IEEE Press, 2018:134-146.
[29] LAU W W, RAMEY N, CORSO J J, et al.Stereo-based endoscopic tracking of cardiac surface deformation[C]//Proceedings of International Conference on Medical Image Computing & Computer-Assisted Intervention-Miccai.Saint-malo, France:[s.n.], 2004:357-368.
[30] STOYANOV D, DARZI A, YANG G Z.Dense 3D depth recovery for soft tissue deformation during robotically assisted laparoscopic surgery[C]//Proceedings of International Conference on Medical Image Computing & Computer-Assisted Intervention-Miccai.Saint-malo, France:[s.n.], 2004:587-596.
[31] STOYANOV D, MYLONAS G P, DELIGIANNI F, et al.Soft-tissue motion tracking and structure estimation for robotic assisted MIS procedures[J].Medical Image Computing and Computer-Assisted Intervention, 2005, 8(2):139-146.
[32] STOYANOV D, SCARZANELLA M V, PRATT P, et al.Real-time stereo reconstruction in robotically assisted minimally invasive surgery[C]//Proceedings of IEEE International Conference on Medical Image Computing & Computer-assisted Intervention.Washington D.C., USA:IEEE Press, 2010:387-396.
[33] STOYANOV D.Stereoscopic scene flow for robotic assisted minimally invasive surgery[C]//Proceedings of IEEE International Conference on Medical Image Computing & Computer-assisted Intervention.Washington D.C., USA:IEEE Press, 2012:578-587.
[34] CHANG P L, STOYANOV D, DAVISON A J, et al.Real-time dense stereo reconstruction using convex optimisation with a cost-volume for image-guided robotic surgery[C]//Proceedings of IEEE International Conference on Medical Image Computing & Computer-Assisted Intervention.Washington D.C., USA:IEEE Press, 2013:588-598.
[35] TOTZ J, THOMPSON S, STOYANOV D, et al.Fast semi-dense surface reconstruction from stereoscopic video in laparoscopic surgery[C]//Proceedings of IEEE International Conference on Information Processing in Computer-Assisted Interventions.Washington D.C., USA:IEEE Press, 2014:763-777.
[36] PENZA V, ORTIZ J, MATTOS L S, et al.Dense soft tissue 3D reconstruction refined with super-pixel segmentation for robotic abdominal surgery[J].International Journal of Computer Assisted Radiology and Surgery, 2016, 11(2):197-206.
[37] SDIRI B, KAANICHE M, CHEIKH F A, et al.Efficient enhancement of stereo endoscopic images based on joint wavelet decomposition and binocular combination[J].IEEE Transactions on Medical Imaging, 2019, 38(1):33-45.
[38] WANG D, LIU H, CHENG X.A miniature binocular endoscope with local feature matching and stereo matching for 3D measurement and 3D reconstruction[J].Sensors, 2018, 18(7):2243.
[39] 王博闻.基于ORB-SLAM2的医学影像三维重建[D].太原:中北大学, 2019. WANG B W.3D reconstruction of medical images based on ORB-SLAM2[D].Taiyuan:North University of China, 2019.(in Chinese)
[40] WANG Z.Deep convolutional network for stereo depth mapping in binocular endoscopy[J].IEEE Access, 2020, 7:12421-12432.
[41] 王国珲, 韩九强, 张新曼, 等.一种新的基于从明暗恢复形状的月球表面三维形状恢复算法[J].宇航学报, 2009, 30(6):2265-2269. WANG G H, HAN J Q, ZHANG X M, et al.A new three-dimensional reconstruction algorithm of the lunar surface based on shape from shading method[J].Journal of Astronautics, 2009, 30(6):2265-2269.(in Chinese)
[42] HORN B K P.Height and gradient from shading[J].International Journal of Computer Vision, 1990, 5(1):37-75.
[43] 王国珲, 王建, 孙帅.SFS三维重构技术的发展现状与趋势研究[J].光学仪器, 2011, 33(4):90-94. WANG G H, WANG J, SUN S.Research on development status and trend of SFS-based 3D reconstruction technique[J].Optical Instruments, 2011, 33(4):90-94.(in Chinese)
[44] OKATANI T, DEGUCHI K.Shape reconstruction from an endoscope image by Shape from shading technique for a point light source at the projection[J].Computer Vision and Image Understanding, 1997, 66(2):19-131.
[45] TANKUS A, SOCHEN N, YESHURUN Y.Reconstruction of medical images by perspective shape-from-shading[C]//Proceedings of the 17th International Conference on Pattern Recognition.Washington D.C., USA:IEEE Press, 2004:158-269.
[46] LI J, REN A M, ZHANG J N.3D reconstruction by perspective shape from shading using linearized triangular element surface model[C]//Proceedings of 2006 International Conference on Mechatronics and Automation.Harbin, China:[s.n.], 2006:378-386.
[47] LAMATA P, MORVAN T, REIMERS M, et al.Addressing shading-based laparoscopic registration[C]//Proceedings of World Congress on Medical Physics and Biomedical Engineering.Munich, Germany:[s.n.], 2009:235-246.
[48] CIUTI G, VISENTINI-SCARZANELLA M, DORE A, et al.Intra-operative monocular 3D reconstruction for image-guided navigation in active locomotion capsule endoscopy[C]//Proceedings of IEEE RAS and EMBS International Conference on Biomedical Robotics and Biomechatronics.Washington D.C., USA:IEEE Press, 2012:768-774.
[49] VISENTINI-SCARZANELLA M, STOYANOV D, YANG G Z.Metric depth recovery from monocular images using Shape-from-shading and specularities[C]//Proceedings of IEEE International Conference on Image Processing.Washington D.C., USA:IEEE Press, 2013:457-468.
[50] TOKGOZOGLU H N, MEISNER E M, KAZHDAN M, et al.Color-based hybrid reconstruction for endoscopy[C]//Proceedings of Computer Vision & Pattern Recognition Workshops.Washington D.C., USA:IEEE Press, 2016:92-101.
[51] LEI L, LI J H, LIU M Q, et al.Shape from shading and optical flow used for 3-dimensional reconstruction of endoscope image[J].Acta Oto-Laryngologica, 2016, 23:1190-1192.
[52] GONCALVES N, ROXO D, BARRETO J, et al.Perspective shape from shading for wide-FOV near-lighting endoscopes[J].Neurocomputing, 2015, 150:136-146.
[53] WU C Y, NARASIMHAN S G, JARAMAZ B.A multi-image shape-from-shading framework for near-lighting perspective endoscopes[J].International Journal of Computer Vision, 2010, 86(2/3):211-228.
[54] TURAN M, PILAVCI Y Y, GANIYUSUFOGLU I, et al.Sparse-then-dense alignment-based 3D map reconstruction method for endoscopic capsule robots[J].Machine Vision and Applications, 2018, 29(2):345-359.
[55] SMITH R, SELF M, CHEESEMAN P.A stochastic map for uncertain spatial relationships[C]//Proceedings of the 4th IEEE International Symposium on Robotics Research.Washington D.C., USA:IEEE Press, 1988, 29:467-474.
[56] 瑚琦, 蔡文龙, 卢定凡, 等.单目视觉惯性的同时定位与地图构建综述[J].软件导刊, 2020, 19(7):275-280. HU Q, CAI W L, LU D F, et al.Survey on monocular visual inertial SLAM algorithms[J].Software Guide, 2020, 19(7):275-280.(in Chinese)
[57] MOUNTNEY P, STOYANOV D, DAVISON A, et al.Simultaneous stereoscope localization and soft-tissue mapping for minimal invasive surgery[C]//Proceedings of IEEE International Conference on Medical Image Computing & Computer-Assisted Intervention.Washington D.C., USA:IEEE Press, 2006:347-356.
[58] DAVISON A J, REID I D, MOLTON N D, et al.MonoSLAM:real-time single camera SLAM[J].IEEE Transactions on Pattern Analysis and Machine Intelligence, 2007, 29(6):1052-1067.
[59] GRASA O G, CIVERA J, MONTIEL J M M.EKF monocular SLAM with relocalization for laparoscopic sequences[C]//Proceedings of IEEE International Conference on Robotics and Automation.Washington D.C., USA:IEEE Press, 2011:4816-4821.
[60] GRASA Ó G, BERNAL E, CASADO S, et al.Visual SLAM for handheld monocular endoscope[J].IEEE Transactions on Medical Imaging, 2014, 33(1):135-146.
[61] MUR-ARTAL R, MONTIEL J M M, TARDOS J D.ORBSLAM:a versatile and accurate monocular SLAM system[J].IEEE Transactions on Robotics, 2015, 31(5):1147-1163.
[62] 揭云飞, 王峰, 智凯旋, 等.视觉SLAM系统分析[J].电脑知识与技术, 2018, 14(19):227-229. JIE Y F, WANG F, ZHI K X, et al.An system analysis of visual SLAM[J].Computer Knowledge and Technology, 2018, 14(19):227-229.(in Chinese)
[63] LIN B, SUN Y, SANCHEZ J E, et al.Efficient vessel feature detection for endoscopic image analysis[J].IEEE Transactions on Bio-Medical Engineering, 2015, 62(4):1141-1150.
[64] 杨滨豪.基于单目SLAM的内窥镜定位和胃肠特征地图构建[D].南昌:南昌大学, 2016. YANG B H.Gastrointestinal endoscope localization and feature map reconstruction based on monocular SLAM[D].Nanchang:Nanchang University, 2016.(in Chinese)
[65] MAHMOUD N, COLLINS T, HOSTETTLER A, et al.Live tracking and dense reconstruction for handheld monocular endoscopy[J].IEEE Transactions on Medical Imaging, 2019, 38(1):79-89.
[66] XIE C, YAO T, WANG J, et al.Endoscope localization and gastrointestinal feature map construction based on monocular SLAM technology[J].Journal of Infection and Public Health, 2020, 13(9):1314-1321.
[67] PICCINELLI N, ROBERTI A, TAGLIABUE E, et al.Rigid 3D registration of pre-operative information for semi-autonomous surgery[C]//Proceedings of 2020 IEEE International Symposium on Medical Robotics.Washington D.C., USA:IEEE Press, 2020:323-336.
[68] LAMARRCA J, PARASHAR S, BARTOLI A, et al.DefSLAM:tracking and mapping of deforming scenes from monocular sequences[J].IEEE Transactions on Robotics, 2020, 58(1):1-13.
[69] 林育斌.基于计算机视觉的三维重建技术研究[D].天津:天津大学, 2010. LIN Y B.Research of 3D reconstruction based on computer vision[D].Tianjin:Tianjin University, 2010.(in Chinese)
[70] 向龙海.基于特征的软组织3D视觉重建与跟踪[D].成都:电子科技大学, 2019. XIANG L H.Feature based 3D visual reconstruction and tracking of soft tissue[D].Chengdu:University of Electronic Science and Technology of China, 2019.(in Chinese)
[71] MORAVEC H P.Rover visual obstacle avoidance[C]//Proceedings of IEEE IJCAIʼ81.Washington D.C., USA:IEEE Press, 1981:785-790.
[72] HARRIS C, STEPHENS M.A combined corner and edge detector[EB/OL].[2021-04-01].http://citeseerx.ist.psu.edu/.
[73] HAN J H, POSTON T T.Chord-to-point distance accumulation and planar curvature:a new approach to discrete curvature[J].Pattern Recognition Letters, 2001, 22(10):1133-1144.
[74] 赵晓芳, 林土胜.视网膜血管图像特征点自动提取和分类[J].计算机工程与应用, 2011, 47(8):14-17. ZHAO X F, LIN T S.Automatic detection and classification of feature points from retinal fundus images[J].Computer Engineering and Applications, 2011, 47(8):14-17.(in Chinese)
[75] LOWE D G.Distinctive image features from scale-invariant keypoints[J].International Journal of Computer Vision, 2004, 60(2):91-110.
[76] BAY H, ESS A, TUYTELAARS T, et al.Speeded-up robust features[J].Computer Vision and Image Understanding, 2008, 110(3):346-359.
[77] 卢鑫.视网膜血管识别技术研究与算法实现[D].成都:电子科技大学, 2017. LU X.Research and implementation on retinal vessel recognition technology[D].Chengdu:University of Electronic Science and Technology of China, 2017.(in Chinese)
[78] CALONDER M, LEPETIT V, ÖZUYSAL M, et al.BRIEF:computing a local binary descriptor very fast[J].IEEE Transactions on Pattern Analysis & Machine Intelligence, 2012, 34(7):1281-1298.
[79] RUBLEE E, RABAUD V, KONOLIGE K, et al.ORB:an efficient alternative to SIFT or SURF[C]//Proceedings of IEEE International Conference on Computer Vision.Washington D.C., USA:IEEE Press, 2011:6-13.
[80] 赵晓芳.视网膜血管图像处理的若干关键问题研究[D].广州:华南理工大学, 2011. ZHAO X F.Research on some key problems in processing of retinal vessel image[D].Guangzhou:South China University of Technology, 2011.(in Chinese)
[81] LIN B, YU S, SANCHEZ J, et al.Vesselness based feature extraction for endoscopic image analysis[C]//Proceedings of the 11th IEEE International Symposium on Biomedical Imaging.Washington D.C., USA:IEEE Press, 2014:581-597.
[82] LIN B X.Visual SLAM and surface reconstruction for abdominal minimally invasive surgery[D].Tampa, USA:University of South Florida, 2015.
[83] ZHANG T, WU K Z, SONG J W, et al.Convergence and consistency analysis for a 3D invariant-EKF SLAM[J].IEEE Robotics & Automation Letters, 2017, 2(2):1158-1169.

选择文件类型/文献管理软件名称

选择包含的内容