Augmented reality: the smart way to guide robotic urologic surgery

3D AR guidance during robot-assisted radical prostatectomy

One of the first AR experiences during robot-assisted radical prostatectomy (RARP) was reported in 2013 by Thompson et al.^[10]. They built and evaluated an intraoperative navigation system overlaying T2-weighted MRI images during surgery. After a preclinical assessment, this technology was used during 13 standard RARPs, evaluating the surgeon’s feedback.

Some years later, Porpiglia et al. published their initial AR experience by using a specifically developed software, named pViewer, in which the 3D models were loaded on a dedicated AR platform^[11]. In this way, it was possible to reproduce the displacements of the target organ during the surgery. To improve the aid given to the surgeon, it was possible to modify the transparency of the 3D model to show the hidden anatomical details of the organ or the tumor’s margins. Thereafter, the AR platform combined the receiving stream from the endoscopic video of the robotic camera with the 3D model images, and the resulting stream was sent back to the remote console using Tile-pro™. Finally, on the console screen were projected the original intraoperative images and the overlapped images, which were continuously moved and adjusted by a dedicated assistant. In this preliminary study, 16 patients undergoing full nerve-sparing RARP for a stage cT2 prostate cancer were compared with 14 patients with cT3 tumors (assessed at MRI) who underwent standard RARP with selective bundles biopsies at the level of suspected extra-prostatic extension. Overall, the positive surgical margin (PSM) rate was 30%, but no PSM was found in pT2 tumors. Selective bundles biopsies performed with AR assistance confirmed extracapsular disease in 11/14 patients (78%). These preliminary data were then confirmed in an update published in 2019^[12]. In this study, the selective biopsies at the level of the neurovascular bundle correctly identified the extra-prostatic extension in 75% of cases.

To improve the 3D models’ realism and organ deformations occurring during the intervention [e.g., during the nerve-sparing (NS) phase], the same group developed a new 3D elastic model^[13]. By using non-linear parametric deformation along the two main axes, these new models were able to replicate the deformation of the prostate during the surgical procedure. The accuracy of this technology was tested by the positioning of a metal clip on the prostate surface at the level of the target lesion, identified thanks to the elastic 3D model in an AR setting. In all patients, the lesion was correctly identified by the model with 100% accuracy.

In 2021, Schiavina et al. reported their experience with the 3D AR guidance during RARP to intraoperatively modulate the nerve-sparing phase of the procedure. 3D AR technology was found to be useful for improving the real-time identification of the index lesion and the subsequent modulation of the nerve-sparing approach in approximately 30% of cases^[14]. Positive surgical margins rate at the level of index lesion was 11.5%, and the erectile function recovery was 65% at six months in patients referred to NS RARP. Moreover, the choice to move towards a more conservative nerve-sparing under 3D AR guidance was revealed to be safe (no PSM) in more than 83% of cases.

3D AR guidance during robot-assisted partial nephrectomy

There is a growing need for a nephron-sparing surgery, claimed for a progressive exploration of 3D AR assistance role by several groups, with the intention to assess its help during the surgical intraoperative navigation.

The impact on the decision-making process was recently evaluated by Wake et al., who reported their experience combining 3D printing and the AR method for image visualization, performing 15 robot-assisted partial nephrectomy (RAPN)^[15]. They showed the value of this tool, giving a proper and safe lead to surgical planning. The major limit, as described by the authors, was the need for diverting the attention of the main surgeon while consulting the model on an external device (e.g., Microsoft HoloLens).

In 2020, Porpiglia et al. published a comparative case-control study evaluating 3D AR guidance vs. intraoperative ultrasound (US) guidance^[16]. In this case, the virtual model was manually overlapped by a dedicated assistant to the endoscopic view of the da Vinci robotic console and displayed using Tile-pro™. In this way, the surgeon remained focused on the surgery without distractions. In total, 91 patients burdened with highly complex renal masses were included in this study (48 in the 3D AR group and 43 in the US group). The 3D AR assistance was more effective than standard intraoperative US, with higher rates of selective clamping and pure enucleation techniques (62.5% vs. 37.5%, P = 0.02). Moreover, the 3D group showed lower collecting system violation rate (10.4% vs. 45.5%, P < 0.001). Concerning the functional assessment, the US group registered a higher reduction of renal function compared to the 3D group, as assessed by renal scan (P = 0.01).

A 15-patient study carried out by Schiavina et al. rigorously reproduced both AR technology and the study design in performing 3D AR RAPN^[17]. Their findings confirm the important role of 3D models in improving the comprehension of vascular anatomy, allowing the surgeon to perform a selective or super-selective clamping in 80% of cases. These choices were concordant in 87% of cases with the preoperative planning depending on the 3D model’s consultation.

The value and impact of a 3D virtual reconstruction-led RAPN have been well examined by two studies from Kobayashi et al. The first study^[18] reported the authors’ first experience with a surgical navigation (SN) system able to overlap and integrate endoscopic images with 3D virtual models while performing RAPNs. Particular attention was given to the skills of two expert surgeons on the handling of the renal artery using (SN group, 32 patients) and not using (non-SN group, 21 patients) the system. The results show how this technology significantly improved the dissection time (16 vs. 9 min in non-SN and SN groups, respectively, P < 0.001) as well as the reduction of inefficient motions (“insert”, “pull”, and “rotate”, all P < 0.03). Furthermore, the improvement in connected motions was positively associated with SN for both surgeons.

The second study^[19] was a retrospective analysis of RAPNs performed from 2013 to 2018 with (102 patients) and without (44 patients) the help of the SN system. This study aimed to evaluate the feasibility of image-guided surgery during RAPN and the association between SN and the renal parenchyma volume after surgery. The final grouping resulted in 42 patients for each arm. The use of SN showed higher parenchymal volume preservation rates (90.0% vs. 83.5%, P = 0.042), thus demonstrating a potential preservation of the renal function after surgery.

Furthermore, Minchiels et al. discussed the importance of 3D virtual reconstructions during nephron-sparing surgery^[20]. Their wide multicenter score-matched analysis included 645 patients who underwent RAPN, of which 230 were performed under 3D guidance. Notwithstanding a larger tumor size and higher surgical complexity in the 3D group, the 3D model guidance resulted in a lower rate of major postoperative complications (3.8% vs. 9.5%), lower estimated glomerular filtration rate (eGFR) variation (-5.6% vs. -10.5%), and higher rate of trifecta (i.e., negative surgical margins, 90% eGFR preservation, and absence of perioperative issues) achievement (55.7% vs. 45.1%). Despite the huge number of 3D guided procedures with a control group included by the authors, which is definitely a strength of the study, their results might mislead to a limited generalization since they analyzed procedures from three expert high-volume surgeons, and there was a lack of standardization in perioperative management among the centers.

New perspectives: automatic guidance with 3D virtual reconstructions

As experienced before, the primary limit of 3D technology can be found in the need for a dedicated assistant to control the precise overlapping. Some scenarios have been proposed to overcome this limit. An interesting idea was offered by Kavoussi et al.^[21]. They developed a touch-based registration, allowing them to automatically anchor the 3D model. This setup was able to track intraoperatively the tip of a robotic gear touching the surface of the kidney, allowing the creation of a 3D surface point set that could be matched to a previously prepared patient-specific model. The authors performed RAPN with their intraoperative navigation system on two phantom kidneys, reproducing on one an exophytic tumor and on the other an endophytic tumor. Landmarks were marked on the phantom organ using a surgical pen and then identified with the robotic manipulator after the initial registration. The 3D model could be shown to the surgeon on the console during the resection, owing to a purposeful picture-in-picture visualization system. The re-registration made after the resection was conducted by the re-identification of the landmarks. Moreover, the difference between the registration made before and after the resection was compared. The average RMS (root mean square) of TRE (target registration error) was, respectively, 2.53 and 4.88 mm in endophytic and exophytic phantoms, confirming a good accuracy of this technique. Furthermore, the resections following preplanned contours were made possible thanks to the intraoperative leading system. After the resection, the re-registration was conducted with small effect on TRE.

Amparore et al. aimed to develop a fully automated overlapping of 3D reconstructions using AR. As well documented in their preliminary experience^[22], they followed the path of computer vision algorithms, for the identification of fiducial points that could be matched on virtual models. Because of the high homogeneity of colors of the operative field, the authors decided to use fluorescence imaging technology [near-infrared fluorescence (NIRF) Firefly system, integrated with the da Vinci platform]. In this way, the whole kidney, appearing as a bright green figure surrounded by a dark field, became the marker for the software. Once the kidney was detected, the software automatically overlapped the 3D model over the endoscopic view [Figure 2]. The kidney’s rotation was calculated, seeing the clear capsular curve or, at least, a portion of it, completed by the virtual borders overlapped on the detected curve. An experienced assistant could refine the matching, when necessary, using a mouse.

Figure 2. Indocyanine green-guided automatic augmented reality RAPN. With the kidney fully exposed from the perirenal fat, a solution of ICG is injected intravenously, allowing the Firefly camera to visualize the kidney, which appears as a bright green organ surrounded by the dark operative field. Subsequently, dedicated software based on computer vision technology automatically anchors the 3D virtual model to the real organ. RAPN: Robot-assisted radical prostatectomy; ICG: indocyanine green; 3D: three-dimensional.

As soon as the registration was achieved, the surgeon could switch to normal vision, maintaining the superimposition of the 3D model over the real anatomy, and proceed with the intervention, i.e., the resection of the renal mass. Of note, all the overlaps executed were correct. Notwithstanding the only preliminary experience, the trial showed promising results for the advent and use of “automatic” guidance in AR.

In the near future, the application of deep learning algorithms and artificial intelligence could represent a step further in enhancing AR guidance^[23,24]. Thanks to their ability to manage a large number of data, the quality assessment of geometrical features for 3D recognition may improve considerably, as already reported in other settings^[25].

These new fields of research could therefore represent the turning point in the surgical intraoperative navigation by improving accuracy and precision of 3D AR guidance.

Previous 3D AR guidance reviews

Some authors have already performed systematic and non-systematic reviews in this field. In 2018, Bertolo et al. reported the first available experiences with AR technology in different urological settings^[26]. Similarly, Roberts et al. recently reviewed the different applications of AR according to the type of surgical procedures, reporting increased registration precision and ability to detect organs boundaries and better surgical outcomes in selected sceneries^[27]. The novelty of our review relies on its subtopic: robotic surgery. In fact, our aim is to provide a comprehensive review of the current applications of AR in the most innovative surgical approach, reporting the available strategies to increase the accuracy of surgical robotic procedures and perioperative outcomes.