Ahead of the game: AR and VR in neurosurgery

3D Simulation of Pituitary crani coronal in captiview
patient-specific 3D rendering generated from fused CT and MRI scans used for planning of a craniotomy to debulk a tumour

Mount Sinai’s Joshua Bederson may have recently hit the headlines for being the first in the world to use an AR image injection system called CaptiView in surgery, but the neurosurgeon has long been involved with bringing technology innovation to the operating room, as Charlotte Ashley finds out.

"As soon as I started doing research around 30 years ago on animal models of subarachnoid haemorrhage [bleeding on the brain’s surface] and brain edema [swelling] MRI scans it quickly became clear that almost any investigation we did would depend in some way on technology,” says Joshua Bederson, reflecting on a career that saw him appointed professor and chairman of Neurosurgery at the Mount Sinai Health System eight years ago, and grace the New York Times’ list of ‘Super Doctors.’

Outside of performing skull-based and cerebrovascular surgeries (including tumour and aneurysm treatment) at Mount Sinai’s Manhattan faculty, Bederson dedicates time to cultivating relationships with vendors that can help improve treatment and learning at the hospital. He says what propels a healthcare provider of Mount’s Sinai’s status (ranked in the top 15 ‘Best Hospitals Rankings Honour Roll’ by the U.S. News & World Report 2016-17) is surgeons becoming advocates for technology across the ten different service lines within neurosurgery.

Dr Joshua Bederson, neurosurgeon at Mount Sinai Hospital headshotBederson recently became the first person in neurosurgery to use a ‘CaptiView’ system that overlays data and 3D models to the eyepiece of a surgical microscope, bringing together relatively rare technologies that are being used individually by only a “handful of experts.” Before the image-guided software was connected to microscope hardware, surgeons would have look away to a monitor to track their movement. The technology is able to track the microscope’s positioning and user’s gaze to autofocus, and gives them the ability to change from live and pre-operative anatomical images via handle controls or a footswitch.

The end product will integrate 3D simulations and AR from company Surgical Theatre with the heads-up display and VR technology from surgical navigation business Brainlab and eyepieces from Leica. “I made a simple proposal [to the manufacturers] that we already have surgical navigation, and the concept of a heads-up display already exists on other platforms - what we want to do now is link surgical navigation into the heads-up display and tie it to the 3D simulation,” says Bederson.

The key element to this, the simulation, involves co-registering and integrating MRI scans, CT scans, angiograms and ultrasound to render the patient’s anatomy in a VR scenario. In the operating room the 3D simulation and patient’s real-life anatomy are linked so that using an optical camera the navigation machine tracks where the tip of a neurosurgeon’s instrument is in relation to blood vessels, structures and a brain tumour.

The introduction of this technology is just one of the applications of Mount Sinai’s Neurosurgery Simulation Core programme led by Bederson, following the success of the hospital’s Virtual Reality Simulation Programme. Started in 2012, it brings together students, residents, neurosurgeons and researchers to work towards improving outcomes and reducing complications in brain surgery with simulation and visualisation technologies.

“[VR] appears to raise the confidence and decrease the fear factor.”

This programme has also led to the advent of the simulation-based consultation at the hospital, using a system currently only found at Mount Sinai according to the system’s vendors. Patients now send their film to the hospital, where a simulation is created so doctors can show them their specific anatomy (such as where a tumour is located in relation to their face and head) on a 3D reconstruction of the patient – something that has brought reassurance to both sides of the consultation room. “Often it’s a ‘aha’-type of moment for the patient, who may have heard from many other doctors ‘You’ve got a tumour that’s located deep in your brain and here’s how we’re going to get there,’ but when you can show them the virtual reality craniotomy and where we’re going to make the opening, it seems they finally understand what we’ve been talking about in a way that’s very different.” He adds: “It appears to raise the confidence and decrease the fear factor.”

Virtual technology has become increasingly commonplace in neurosurgery over the past 10-15 years. Mount Sinai’s Neurosurgery department was the first in the country to acquire a NeuroTouch VR simulator, which uses 3D software and handheld surgical control to provide visual and haptic feedback to mimic a real-life brain surgery, and has developed its own software for Google Glass that can record surgical performances. Yet the huge recent surge of interest in AR and VR has caused the medical world to really take notice. “There’s so much interest in virtual reality now that patients and doctors are all clamouring to do it,” says Bederson.

This attention doesn’t always convert into enthusiasm, however. “There is enormous scepticism on the part of established surgeons. One of the things you learn your whole surgical life is to convert the information you see on X-rays into your own 3D concept – that’s what makes a good surgeon, particularly in neurosurgery where the anatomy is so complicated.” He continues; “A good surgeon is able to apply that knowledge to what they see through the operating microscope, and this technology may be seen by some as a crutch, or a way of avoiding having to incorporate that knowledge into your skillset.”

“What I’ve seen in fact is that it helps you incorporate that knowledge much faster, and it brings people in learning phases up to speed much faster then they might have otherwise. Even for experienced people like me, it is very reassuring, and it allows me to operate faster and with a greater degree of safety.”

As well as managing scepticism from counterparts, neurosurgeons like Bederson must convince their employer of the value of bringing virtual technology into hospitals. “How do I convince the hospital administrator that this US $300,000 or more piece of equipment should be purchased? They ask, ‘Show me the benefit on outcome and the reduced cost that this technology brings,’ and that’s not possible yet, but I do think there’s going to be an overwhelming push in this direction.”

WATCH: See CaptiView technology in action in the operating room.