Robotic Intra-Operative Ultrasound: Virtual Environments and Parallel Systems

Shuangyi Wang; James Housden; Tianxiang Bai; Hongbin Liu; Junghwan Back; Davinder Singh; Kawal Rhode; Zeng-Guang Hou; Fei-Yue Wang

doi:10.1109/JAS.2021.1003985

Volume 8 Issue 5

May 2021

IEEE/CAA Journal of Automatica Sinica

JCR Impact Factor: 11.8, Top 4% (SCI Q1)

CiteScore: 17.6, Top 3% (Q1)
Google Scholar h5-index: 77， TOP 5

Turn off MathJax

Article Contents

Article Navigation > IEEE/CAA Journal of Automatica Sinica > 2021 > 8(5): 1095-1106

S. Y. Wang, J. Housden, T. X. Bai, H. B. Liu, J. Back, D. Singh, K. Rhode, Z.-G. Hou, and F.-Y. Wang, "Robotic Intra-Operative Ultrasound: Virtual Environments and Parallel Systems," IEEE/CAA J. Autom. Sinica, vol. 8, no. 5, pp. 1095-1106, May. 2021. doi: 10.1109/JAS.2021.1003985

Citation:

S. Y. Wang, J. Housden, T. X. Bai, H. B. Liu, J. Back, D. Singh, K. Rhode, Z.-G. Hou, and F.-Y. Wang, "Robotic Intra-Operative Ultrasound: Virtual Environments and Parallel Systems," IEEE/CAA J. Autom. Sinica, vol. 8, no. 5, pp. 1095-1106, May. 2021. doi: 10.1109/JAS.2021.1003985

Citation:

S. Y. Wang, J. Housden, T. X. Bai, H. B. Liu, J. Back, D. Singh, K. Rhode, Z.-G. Hou, and F.-Y. Wang, "Robotic Intra-Operative Ultrasound: Virtual Environments and Parallel Systems," IEEE/CAA J. Autom. Sinica, vol. 8, no. 5, pp. 1095-1106, May. 2021. doi: 10.1109/JAS.2021.1003985

PDF( 1662 KB)

Robotic Intra-Operative Ultrasound: Virtual Environments and Parallel Systems

doi: 10.1109/JAS.2021.1003985

1.
State Key Laboratory of Management and Control of Complex Systems, Chinese Academy of Sciences, Institute of Automation, Beijing 100190, China
2.
School of Biomedical Engineering and Imaging Sciences, King’s College London, London SE1 7EH, UK
3.
Xtronics, Ltd., Gravesend DA12 2AD, UK

Funds: This work was supported in part by the Key Research and Development Program 2020 of Guangzhou (202007050002), in part by the National Natural Science Foundation of China (62003339, U1811463), and in part by the Intel Collaborative Research Institute for Intelligent and Automated Connected Vehicles (“ICRI-IACV”)

More Information

Author Bio:
Shuangyi Wang (M’20) received the B.Eng. and Ph.D. degrees in biomedical engineering from Tianjin University in 2013 and King’s College London, UK, in 2017, respectively. From 2017 to 2020, he was a Research Associate with the School of Biomedical Engineering and Imaging Sciences at King’s College London. He is currently an Associate Professor at the Institute of Automation, Chinese Academy of Sciences. His research interests include medical robotics, robotic ultrasound, and robotic-assisted surgery

James Housden received the B.A. and M.Eng. in engineering from the University of Cambridge, UK in 2004, and the Ph.D. in ultrasound imaging from the Department of Engineering, University of Cambridge in 2008. He has been employed at the School of Biomedical Engineering and Imaging Sciences, King’s College London, UK, since 2011 and is currently a Senior Teaching Fellow. His research interests include ultrasound imaging, interventional imaging, and biomedical robotics

Tianxiang Bai received the B.S. degree from Zhejiang University in 2013, and he is currently pursuing the Ph.D. degree with the Institute of Automation, Chinese Academy of Sciences. His research interests include machine learning, robotics, human-robot interaction, and robotic art

Hongbin Liu (M’07) is a Reader in medical robotics in the School of Biomedical Engineering and Imaging Sciences, Faculty of Life Sciences and Medicine, King’s College London, UK. He is the Director of Haptics Mechatronics and Medical Robotics (HaMMeR) Laboratory at the Centre for Robotics Research. His research lies in creating soft and compliant robots with appropriate haptic perception to elegantly and safely interact with changing and unstructured environment, in the same time, enhancing the understanding of the environment through the haptic interaction. Applications of the research include haptic sensing of medical instruments, haptic sensing and control for steerable catheters and endoscopes for medical intervention and diagnosis, and robotic ultrasound manipulation

Junghwan Back received the Ph.D. degree in robotics from the Department of Informatics, Faculty of Natural & Mathematical Sciences, King’s College London. Currently, he is a Research Associate in the School of Biomedical Engineering and Imaging Sciences, Faculty of Life Sciences and Medicine, King’s College London. His research interests include robot assisted cardiac catheter ablation and tactile sensing

Davinder Singh is the Founder of Xtronics Ltd. He has worked with robotics, X-ray, and imaging technologies for 40 years. His expertise lies in robotics, real-time X-ray, and Micro CT systems, along with demountable and high energy electron pulse guns. As an OEM, he has delivered bespoke robotic systems for applications ranging from large scale industrial X-ray imaging to medical ultrasonic scanning for BAE Defence systems, Airbus and Rolls Royce. His professional interests include imaging technologies, non-destructive testing, 3D printing, industrial and biomedical robotics. In the last 8 years, he has been working alongside King’s College London, offering expertise and support

Kawal Rhode received the bachelor degree in basic medical sciences and radiological sciences at Guy’s & St. Thomas’ Hospitals Medical School in 1992 and the doctorate in medical physics from the Department of Surgery, University College London, in 2006. Prof. Rhode was employed at the School of Biomedical Engineering and Imaging Sciences, King’s College London, since 2001 and is currently Professor of biomedical engineering and Head of Education. His current research interests include image-guided interventions, intelligent mechatronics systems for interventions and ultrasound imaging, 3D printing in healthcare and pedagogy for biomedical engineering. He has more than 400 publications in journals, conference proceedings, book chapters, and patents

Zeng-Guang Hou (SM’09–F’19) received the B.E. and M.E. degrees in electrical engineering from Yanshan University in 1991 and 1993, respectively, and the Ph.D. degree in electrical engineering from Beijing Institute of Technology in 1997. From May 1997 to June 1999, he was a Postdoctoral Research Fellow in the Key Laboratory of Systems and Control, Institute of Systems Science, Chinese Academy of Sciences. He was a Research Assistant at the Hong Kong Polytechnic University, Hong Kong, China, from May 2000 to January 2001. From July 1999 to May 2004, he was an Associate Professor at the Institute of Automation, Chinese Academy of Sciences, and has been a Full Professor since June 2004. He is now the Deputy Director of the State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences. His research interests include neural networks, robotics, and intelligent systems

Fei-Yue Wang (S’87–M’89–SM’94–F’03) received the Ph.D. degree in computer and systems engineering from Rensselaer Polytechnic Institute, Troy, USA, in 1990. He joined the University of Arizona, Tucson, in 1990 and became a Professor and the Director of the Robotics and Automation Laboratory and the Program in Advanced Research for Complex Systems. In 1999, he founded the Intelligent Control and Systems Engineering Center, Institute of Automation, Chinese Academy of Sciences (CAS). In 2002, he participated in the development of the Key Laboratory of Complex Systems and Intelligence Science, CAS, as the Director, where he was also the Vice President for research, education, and academic exchanges at the Institute of Automation from 2006 to 2010. In 2011, he was named as the Director of the State Key Laboratory for Management and Control of Complex Systems. His current research interests include methods and applications for parallel systems, social computing, and knowledge automation
Corresponding author: Fei-Yue Wang, e-mail: feiyue.wang@ia.ac.cn
Received Date: 2021-01-28
Accepted Date: 2021-02-27

Available Online: 2021-03-15

Abstract

Abstract

Robotic intra-operative ultrasound has the potential to improve the conventional practice of diagnosis and procedure guidance that are currently performed manually. Working towards automatic or semi-automatic ultrasound, being able to define ultrasound views and the corresponding probe poses via intelligent approaches become crucial. Based on the concept of parallel system which incorporates the ingredients of artificial systems, computational experiments, and parallel execution, this paper utilized a recent developed robotic trans-esophageal ultrasound system as the study object to explore the method for developing the corresponding virtual environments and present the potential applications of such systems. The proposed virtual system includes the use of 3D slicer as the main workspace and graphic user interface (GUI), Matlab engine to provide robotic control algorithms and customized functions, and PLUS (Public software Library for UltraSound imaging research) toolkit to generate simulated ultrasound images. Detailed implementation methods were presented and the proposed features of the system were explained. Based on this virtual system, example uses and case studies were presented to demonstrate its capabilities when used together with the physical TEE robot. This includes standard view definition and customized view optimization for pre-planning and navigation, as well as robotic control algorithm evaluations to facilitate real-time automatic probe pose adjustments. To conclude, the proposed virtual system would be a powerful tool to facilitate the further developments and clinical uses of the robotic intra-operative ultrasound systems.
- Medical robots,
- parallel intelligence,
- parallel systems,
- robotic-assisted surgery,
- robotic ultrasound

FullText(HTML)

References(30)

References

[1]	N. Magnavita, L. Bevilacqua, P. Mirk, A. Fileni, and N. Castellino, “Work-related musculoskeletal complaints in sonologists,” J. Occup. Environ. Med., vol. 41, no. 11, pp. 981–988, 1999. doi: 10.1097/00043764-199911000-00010
[2]	L. LaGrone, V. Sadasivam, A. Kushner, and R. Groen, “A review of training opportunities for ultrasonography in low and middle income countries,” Tropical Med. Int. Health, vol. 17, no. 7, pp. 808–819, 2012. doi: 10.1111/j.1365-3156.2012.03014.x
[3]	C. Kim, F. Schäfer, D. Chang, D. Petrisor, M. Han, and D. Stoianovici, “Robot for ultrasound-guided prostate imaging and intervention,” in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst., 2011, pp. 943–948.
[4]	M. Han, C. Kim, P. Mozer, F. Schäfer, S. Badaan, B. S, Vigaru, et al., “Tandem-robot Assisted Laparoscopic Radical Prostatectomy to Improve the Neurovascular Bundle Visualization: A Feasibility Study,” Urology, vol. 77, no. 2, pp. 502–506, 2011.
[5]	T. Adebar, S. Salcudean, S. Mahdavi, M. Moradi, C. Nguan, and L. Goldenberg, “A robotic system for intra-operative trans-rectal ultrasound and ultrasound elastography in radical prostatectomy,” in Proc. Int. Conf. Inf. Process. Comput.-Assisted Interventions, 2011, pp. 79–89.
[6]	T. Adebar, O. Mohareri, and S. Salcudean, “Instrument-based calibration and remote control of intraoperative ultrasound for robot-assisted surgery,” in Proc. Int. Conf. Biomed. Robot. Biomechatronics, 2012, pp. 38–43.
[7]	C. M. Schneider, A. M. Okamura, and G. Fichtinger, “A robotic system for transrectal needle insertion into the prostate with integrated ultrasound,” in Proc. Int. Conf. Robot. Autom., 2004, pp. 365–370.
[8]	T.Podder, I. Buzurovic, K. Yan, Y. Hu, R. Valicenti, A. Dicker, et al., “Robotic system for image-guided prostate seed implant,” Brachytherapy, vol. 7, no. 2, pp. 100–101, 2008.
[9]	P. M. Loschak, L. J. Brattain, and R. D. Howe, “Automated pointing of cardiac imaging catheters,” in Proc. Int. Conf. Robot. Autom., 2013, pp. 5794–5799.
[10]	P. Loschak, A. Degirmenci, Y. Tenzer, C. Tschabrunn, E. Anter, and R. Howe, “A four degree of freedom robot for positioning ultrasound imaging catheters,” J. Mech. Robot, vol. 8, no. 5, 2016.
[11]	S. Wang, J. Housden, D. Singh, K. Althoefer, and K. Rhode, “Design, testing and modelling of a novel robotic system for trans‐oesophageal ultrasound,” Int. J. Medical Robot. Comput. Assist. Surg., vol. 12, no. 3, pp. 342–354, 2016. doi: 10.1002/rcs.1691
[12]	S. Wang, D. Singh, D. Johnson, K. Althoefer, K. Rhode, and R. J. Housden, “Robotic ultrasound: View planning, tracking, and automatic acquisition of transesophageal echocardiography,” IEEE Robot. Autom. Mag., vol. 23, no. 4, pp. 118–127, 2016. doi: 10.1109/MRA.2016.2580478
[13]	J. A. Goldstein, S. Balter, M. Cowley, J. Hodgson, and L. W. Klein, “Occupational hazards of interventional cardiologists: Prevalence of orthopedic health problems in contemporary practice,” Catheter. Cardiovasc. Intervent., vol. 63, pp. 407–411, 2004. doi: 10.1002/ccd.20201
[14]	F.-Y. Wang, “Shadow systems: A new concept for nested and embedded co-simulation for intelligent systems”, University of Arizona, Tucson, USA, 1994.
[15]	F.-Y. Wang, “On the modeling, analysis, control and management of complex systems,” Complex Syst. Complex. Sci., vol. 3, no. 2, pp. 26–34, 2006.
[16]	F.-Y. Wang, “Parallel system methods for management and control of complex systems,” Control and Decision, vol. 19, no. 5, pp. 485–489, 514, 2004.
[17]	F.-Y. Wang, N. N. Zheng, D. P. Cao, C. M. Martinez, L. Li, and T. Liu, “Parallel driving in CPSS: A unified approach for transport automation and vehicle intelligence,” IEEE/CAA J. Autom. Sinica, vol. 4, no. 4, pp. 577–587, 2017. doi: 10.1109/JAS.2017.7510598
[18]	K. F. Wang, C. Gou, N. N. Zheng, J. M. Rehg, and F.-Y. Wang, “Parallel vision for perception and understanding of complex scenes: Methods, framework, and perspectives,” Artif. Intell. Rev., vol. 48, no. 3, pp. 299–329, 2017. doi: 10.1007/s10462-017-9569-z
[19]	J. S. Shanewise, A. T. Cheung, S. Aronson, W. J. Stewart, R. L. Weiss, J. B. Mark, et al., “ASE/SCA guidelines for performing a comprehensive intraoperative multiplane transesophageal echocardiography examination: recommendations of the american society of echocardiography council for intraoperative echocardiography and the society of cardiovascular anesthesiologists task force for certification in perioperative transesophageal echocardiography,” Anesth. & Analg., vol. 89, pp. 870–84, 1999.
[20]	A. Lasso, T. Heffter, A. Rankin, C. Pinter, T. Ungi, and G. Fichtinger, “PLUS: Open-source toolkit for ultrasound-guided intervention systems,” IEEE. Trans. Biomed. Eng., vol. 61, no. 10, pp. 2527–2537, 2014. doi: 10.1109/TBME.2014.2322864
[21]	J. Peters, O. Ecabert, C. Meyer, H. Schramm, R. Kneser, A. Groth, and J.Weese, “Automatic whole heart segmentation in static magnetic resonanceimage volumes,” in Proc. Int. Conf. Med. Image Comput. Comput. Assist. Intervention, Lecture Notes Comput. Sci., 2007, pp. 402–410.
[22]	L. Bartha, A. Lasso, C. Pinter, T. Ungi, Z. Keri, and G. Fichtinger, “Open-source surface mesh-based ultrasound-guided spinal intervention simulator,” Int. J. Comput. Assist. Radiol. Surg., vol. 8, no. 6, pp. 1043–1051, 2013. doi: 10.1007/s11548-013-0901-z
[23]	J. Tokuda, G. S. Fischer, X. Papademetris, Z. Yaniv, L. Ibanez, P. Cheng, et al., “OpenIGTLink: an open network protocol for image-guided therapy environment,” Int. J. Medical Robot. Comput. Assist. Surg., vol. 5, no. 4, pp. 423–434, 2009.
[24]	S. Wang, “Development of a robotic trans-oesophageal ultrasound system and its application in automatic acquisition,” Ph.D. dissertation, School of Biomedical Engineering & Imaging Sciences, King’s College London., London, UK, 2017.
[25]	E. Catmull and R. Rom, “A class of local interpolating splines,” Comput. Aided Geom. Des., pp. 317–326, 1974.
[26]	V. Malik, M. Hote, and A. Jha, “Minimally invasive cardiac surgery and transesophageal echocardiography,” Ann. Card. Anaesth., vol. 17, no. 2, Article No. 125, 2014. doi: 10.4103/0971-9784.129844
[27]	M. Prabhu and A. George, “Transesophageal monitoring in anaesthesia: An update,” Curr. Anesthesiol. Rep., vol. 4, no. 3, pp. 261–273, 2014. doi: 10.1007/s40140-014-0071-8
[28]	K. Deb, “An efficient constraint handling method for genetic algorithms,” Comput. Methods Appl. Mech. Eng., vol. 186, no. 2-4, pp. 311–338, 2000. doi: 10.1016/S0045-7825(99)00389-8
[29]	K. Deep, K. Singh, M. Kansal, and C. Mohan, “A real coded genetic algorithm for solving integer and mixed integer optimization problems,” Appl. Math. Comput., vol. 212, no. 2, pp. 505–518, 2009.
[30]	S. Wang, J. Housden, A. Zar, R. Gandecha, D. Singh, and K. Rhode, “Strategy for monitoring cardiac interventions with an intelligent robotic ultrasound device,” Micromachines, vol. 9, no. 2, Article No. 65, 2018. doi: 10.3390/mi9020065

Supplements(0)

Cited By

Proportional views

Proportional views

通讯作者: 陈斌, bchen63@163.com

1.
沈阳化工大学材料科学与工程学院沈阳 110142

Figures(15)

Get Citation

PDF

XML

Article Metrics

Article views (1816) PDF downloads(54)

Highlights

A virtual environment to represent the use of the robotic trans-esophageal ultrasound is presented.
Example uses are introduced to demonstrate its applications in surgical planning.
Concept of parallel systems is utilized for a specific medical robotic application.
The proposed virtual environment would assist the robotic-based automatic ultrasound.

Robotic Intra-Operative Ultrasound: Virtual Environments and Parallel Systems

doi: 10.1109/JAS.2021.1003985

Abstract

References

Proportional views

Catalog

通讯作者: 陈斌, bchen63@163.com

Article Metrics

Highlights

Export File

Citation

Format

Content