Fusion of augmented reality imaging with the endoscopic view for endonasal skull base surgery; a novel application for surgical navigation based on intraoperative cone beam computed tomography and optical tracking

Autoři: Marco Lai aff001;  Simon Skyrman aff003;  Caifeng Shan aff001;  Drazenko Babic aff001;  Robert Homan aff004;  Erik Edström aff003;  Oscar Persson aff003;  Gustav Burström aff003;  Adrian Elmi-Terander aff003;  Benno H. W. Hendriks aff001;  Peter H. N. de With aff002
Působiště autorů: Philips Research, Eindhoven, The Netherlands aff001;  Eindhoven University of Technology (TU/e), Eindhoven, The Netherlands aff002;  Department of Neurosurgery, Karolinska University Hospital and Department of Clinical Neuroscience, Karolinska Institutet, Stockholm, Sweden aff003;  Philips Healthcare, Best, the Netherlands aff004;  Department of Biomechanical Engineering, Delft University of Technology, Delft, The Netherlands aff005
Vyšlo v časopise: PLoS ONE 15(1)
Kategorie: Research Article
doi: https://doi.org/10.1371/journal.pone.0227312



Surgical navigation is a well-established tool in endoscopic skull base surgery. However, navigational and endoscopic views are usually displayed on separate monitors, forcing the surgeon to focus on one or the other. Aiming to provide real-time integration of endoscopic and diagnostic imaging information, we present a new navigation technique based on augmented reality with fusion of intraoperative cone beam computed tomography (CBCT) on the endoscopic view. The aim of this study was to evaluate the accuracy of the method.

Material and methods

An augmented reality surgical navigation system (ARSN) with 3D CBCT capability was used. The navigation system incorporates an optical tracking system (OTS) with four video cameras embedded in the flat detector of the motorized C-arm. Intra-operative CBCT images were fused with the view of the surgical field obtained by the endoscope’s camera. Accuracy of CBCT image co-registration was tested using a custom-made grid with incorporated 3D spheres.


Co-registration of the CBCT image on the endoscopic view was performed. Accuracy of the overlay, measured as mean target registration error (TRE), was 0.55 mm with a standard deviation of 0.24 mm and with a median value of 0.51mm and interquartile range of 0.39˗˗0.68 mm.


We present a novel augmented reality surgical navigation system, with fusion of intraoperative CBCT on the endoscopic view. The system shows sub-millimeter accuracy.

Klíčová slova:

Cameras – Computed axial tomography – Endoscopic surgery – Endoscopy – Magnetic resonance imaging – Skull – Surgical and invasive medical procedures – Endoscopic plastic surgery


1. Lipski SM, Digonnet A, Dolhen PJE. Modern indications for endoscopic endonasal surgery. 2016;4(1):96–102.

2. Zwagerman NT, Zenonos G, Lieber S, Wang W-H, Wang EW, Fernandez-Miranda JC, et al. Endoscopic transnasal skull base surgery: pushing the boundaries. 2016;130(2):319–30. doi: 10.1007/s11060-016-2274-y 27766473

3. American Academy of Otolaryngology-Head & Neck Surgery. Position Statement: intra-operative use of computer aided surgery. 2014. Available at: http://www.entnet.org/content/intra-operative-use-computer-aided-surgery. Accessed August 20, 2016. 2016.

4. Hepworth EJ, Bucknor M, Patel A, Vaughan WCJOH, Surgery N. Nationwide survey on the use of image-guided functional endoscopic sinus surgery. 2006;135(1):68–75. doi: 10.1016/j.otohns.2006.01.025 16815185

5. Justice JM, Orlandi RR, editors. An update on attitudes and use of image‐guided surgery. International forum of allergy & rhinology; 2012: Wiley Online Library.

6. Orlandi RR, Petersen EJAjor. Image guidance: a survey of attitudes and use. 2006;20(4):406–11. doi: 10.2500/ajr.2006.20.2884 16955769

7. Tabaee A, Kassenoff TL, Kacker A, Anand VKJOH, Surgery N. The efficacy of computer assisted surgery in the endoscopic management of cerebrospinal fluid rhinorrhea. 2005;133(6):936–43. doi: 10.1016/j.otohns.2005.07.028 16360517

8. Dubin MR, Tabaee A, Scruggs JT, Kazim M, Close LGJAoO, Rhinology, Laryngology. Image-guided endoscopic orbital decompression for Graves' orbitopathy. 2008;117(3):177–85. doi: 10.1177/000348940811700304 18444477

9. Tschopp KP, Thomaser EGJR. Outcome of functional endonasal sinus surgery with and without CT-navigation. 2008;46(2):116–20. 18575012

10. Al-Swiahb JN, Al Dousary SHJAoSm. Computer-aided endoscopic sinus surgery: a retrospective comparative study. 2010;30(2):149. doi: 10.4103/0256-4947.60522 20220266

11. Dalgorf DM, Sacks R, Wormald P-J, Naidoo Y, Panizza B, Uren B, et al. Image-guided surgery influences perioperative morbidity from endoscopic sinus surgery: a systematic review and meta-analysis. 2013;149(1):17–29. doi: 10.1177/0194599813488519 23678278

12. Fried MP, Moharir VM, Shin J, Taylor-Becker M, Morrison P, Kennedy DWJAjor. Comparison of endoscopic sinus surgery with and without image guidance. 2002;16(4):193–7. 12222943

13. Javer AR, Genoway KAJJoo. Patient quality of life improvements with and without computer assistance in sinus surgery: outcomes study. 2006;35(6). doi: 10.2310/7070.2006.0083 17380830

14. Masterson L, Agalato E, Pearson CJTJoL, Otology. Image-guided sinus surgery: practical and financial experiences from a UK centre 2001–2009. 2012;126(12):1224–30. doi: 10.1017/S002221511200223X 23067580

15. Metson R, Cosenza M, Gliklich RE, Montgomery WWJAoOH, Surgery N. The role of image-guidance systems for head and neck surgery. 1999;125(10):1100–4. doi: 10.1001/archotol.125.10.1100 10522501

16. Schulze F, Bühler K, Neubauer A, Kanitsar A, Holton L, Wolfsberger SJIjocar, et al. Intra-operative virtual endoscopy for image guided endonasal transsphenoidal pituitary surgery. 2010;5(2):143–54. doi: 10.1007/s11548-009-0397-8 20033497

17. Reardon EJJTL. Navigational risks associated with sinus surgery and the clinical effects of implementing a navigational system for sinus surgery. 2002;112(S99):1–19.

18. Rombaux P, Ledeghen S, Hamoir M, Bertrand B, Eloy P, Coche E, et al. Computer assisted surgery and endoscopic endonasal approach in 32 procedures. 2003;57(2):131–7. 12836470

19. Eliashar R, Sichel J, Gross M, Hocwald E, Dano I, Biron A, et al. Image guided navigation system—a new technology for complex endoscopic endonasal surgery. 2003;79(938):686–90. 14707243

20. Burström G, Nachabe R, Persson O, Edström E, Terander AEJS. Augmented and Virtual Reality Instrument Tracking for Minimally Invasive Spine Surgery: A Feasibility and Accuracy Study. 2019. doi: 10.1097/BRS.0000000000003006 30830046

21. Bong JH, Song Hj, Oh Y, Park N, Kim H, Park SJTIJoMR, et al. Endoscopic navigation system with extended field of view using augmented reality technology. 2018;14(2):e1886.

22. Li L, Yang J, Chu Y, Wu W, Xue J, Liang P, et al. A novel augmented reality navigation system for endoscopic sinus and skull base surgery: a feasibility study. 2016;11(1):e0146996. doi: 10.1371/journal.pone.0146996 26757365

23. Salehahmadi F, Hajialiasgari FJWjops. Grand Adventure of Augmented Reality in Landscape of Surgery. 2019;8(2):135. doi: 10.29252/wjps.8.2.135 31309050

24. Eckert M, Volmerg JS, Friedrich CMJJm, uHealth. Augmented reality in medicine: systematic and bibliographic review. 2019;7(4):e10967. doi: 10.2196/10967 31025950

25. Mikhail M, Mithani K, Ibrahim GMJWn. Presurgical and Intraoperative Augmented Reality in Neuro-oncologic Surgery: Clinical Experiences and Limitations. 2019. doi: 10.1016/j.wneu.2019.04.256 31103764

26. Elmi-Terander A, Burström G, Nachabe R, Skulason H, Pedersen K, Fagerlund M, et al. Pedicle Screw Placement Using Augmented Reality Surgical Navigation with Intraoperative 3D Imaging: A First In-Human Prospective Cohort Study. 2019;44(7):517–25. doi: 10.1097/BRS.0000000000002876 30234816

27. Citardi MJ, Yao W, Luong AJOCoNA. Next-Generation Surgical Navigation Systems in Sinus and Skull Base Surgery. 2017;50(3):617–32. doi: 10.1016/j.otc.2017.01.012 28392037

28. Citardi MJ, Agbetoba A, Bigcas JL, Luong A, editors. Augmented reality for endoscopic sinus surgery with surgical navigation: a cadaver study. International forum of allergy & rhinology; 2016: Wiley Online Library.

29. Mirota DJ, Wang H, Taylor RH, Ishii M, Gallia GL, Hager GDJItomi. A system for video-based navigation for endoscopic endonasal skull base surgery. 2011;31(4):963–76. doi: 10.1109/TMI.2011.2176500 22113772

30. Batra PS, Kanowitz SJ, Citardi MJJAjor. Clinical utility of intraoperative volume computed tomography scanner for endoscopic sinonasal and skull base procedures. 2008;22(5):511–5. doi: 10.2500/ajr.2008.22.3216 18954511

31. Elmi-Terander A, Nachabe R, Skulason H, Pedersen K, Söderman M, Racadio J, et al. Feasibility and accuracy of thoracolumbar minimally invasive pedicle screw placement with augmented reality navigation technology. 2018;43(14):1018. doi: 10.1097/BRS.0000000000002502 29215500

32. Elmi-Terander A, Skulason H, Söderman M, Racadio J, Homan R, Babic D, et al. Surgical navigation technology based on augmented reality and integrated 3D intraoperative imaging: a spine cadaveric feasibility and accuracy study. 2016;41(21):E1303. doi: 10.1097/BRS.0000000000001830 27513166

33. Jackman AH, Palmer JN, Chiu AG, Kennedy DWJAjor. Use of intraoperative CT scanning in endoscopic sinus surgery: a preliminary report. 2008;22(2):170–4. doi: 10.2500/ajr.2008.22.3153 18416975

34. Edström E, Burström G, Nachabe R, Gerdhem P, Elmi-Terander AJON. A Novel Augmented-Reality-Based Surgical Navigation System for Spine Surgery in a Hybrid Operating Room: Design, Workflow, and Clinical Applications. Epub ahead of print, available at: https://doi.org/10.1093/ons/opz236 Accessed Augsut 27 2019. doi: 10.1093/ons/opz236

35. Zhang ZJITopa, intelligence m. A flexible new technique for camera calibration. 2000;22.

36. Lai M, Shan C, editors. Hand-eye camera calibration with an optical tracking system. Proceedings of the 12th International Conference on Distributed Smart Cameras; 2018: ACM.

37. Gao X-S, Hou X-R, Tang J, Cheng H-FJItopa, intelligence m. Complete solution classification for the perspective-three-point problem. 2003;25(8):930–43.

38. Gumprecht HK, Widenka DC, Lumenta CB. Brain Lab VectorVision neuronavigation system: technology and clinical experiences in 131 cases. Neurosurgery. 1999;44(1):97–104. doi: 10.1097/00006123-199901000-00056 9894969

39. Labadie RF, Davis BM, Fitzpatrick JMJCoio, head, surgery n. Image-guided surgery: what is the accuracy? 2005;13(1):27–31. doi: 10.1097/00020840-200502000-00008 15654212

40. Schlaier J, Warnat J, Brawanski AJCAS. Registration accuracy and practicability of laser-directed surface matching. 2002;7(5):284–90. doi: 10.1002/igs.10053 12582981

41. Snyderman C, Zimmer LA, Kassam AJOH, Surgery N. Sources of registration error with image guidance systems during endoscopic anterior cranial base surgery. 2004;131(3):145–9. doi: 10.1016/j.otohns.2004.03.002 15365528

42. Mirota DJ, Uneri A, Schafer S, Nithiananthan S, Reh DD, Ishii M, et al. Evaluation of a system for high-accuracy 3d image-based registration of endoscopic video to c-arm cone-beam ct for image-guided skull base surgery. 2013;32(7):1215–26. doi: 10.1109/TMI.2013.2243464 23372078

43. Winne C, Khan M, Stopp F, Jank E, Keeve EJIjocar, surgery. Overlay visualization in endoscopic ENT surgery. 2011;6(3):401–6. doi: 10.1007/s11548-010-0507-7 20577827

44. Daly MJ, Chan H, Nithiananthan S, Qiu J, Barker E, Bachar G, et al., editors. Clinical implementation of intraoperative cone-beam CT in head and neck surgery. Medical Imaging 2011: Visualization, Image-Guided Procedures, and Modeling; 2011: International Society for Optics and Photonics.

45. Daly MJ, Chan H, Prisman E, Vescan A, Nithiananthan S, Qiu J, et al., editors. Fusion of intraoperative cone-beam CT and endoscopic video for image-guided procedures. Medical Imaging 2010: Visualization, Image-Guided Procedures, and Modeling; 2010: International Society for Optics and Photonics.

46. Hamming NM, Daly MJ, Irish JC, Siewerdsen JH. Automatic image-to-world registration based on x-ray projections in cone-beam CT-guided interventions. Med Phys. 2009;36(5):1800–12. Epub 2009/06/24. doi: 10.1118/1.3117609 19544799; PubMed Central PMCID: PMC2832033.

47. Prisman E, Daly MJ, Chan H, Siewerdsen JH, Vescan A, Irish JC, editors. Real‐time tracking and virtual endoscopy in cone‐beam CT‐guided surgery of the sinuses and skull base in a cadaver model. International forum of allergy & rhinology; 2011: Wiley Online Library.

48. Burström G, Buerger C, Hoppenbrouwers J, Nachabe R, Lorenz C, Babic D, et al. Machine learning for automated 3-dimensional segmentation of the spine and suggested placement of pedicle screws based on intraoperative cone-beam computer tomography. 2019;1(aop):1–8.

49. Dixon BJ, Daly MJ, Chan HH, Vescan A, Witterick IJ, Irish JCJAjor, et al. Inattentional blindness increased with augmented reality surgical navigation. 2014;28(5):433–7. doi: 10.2500/ajra.2014.28.4067 25198032

50. Yeh M, Wickens CDJHF. Display signaling in augmented reality: Effects of cue reliability and image realism on attention allocation and trust calibration. 2001;43(3):355–65. doi: 10.1518/001872001775898269 11866192

Článek vyšel v časopise


2020 Číslo 1
Nejčtenější tento týden