Real-time 3D ultrasound in augmented reality helps novices perform like experts

Interpreting medical ultrasound images is a difficult task, requiring a technician to look at 2D images and mentally arrange them into a 3D representation of what the tissue looks like.

To make that job easier, MIT researchers developed a new approach to ultrasound imaging that allows the user to visualize a 3D augmented-reality image of the object being scanned. Using a virtual-reality headset, they can see a precise 3D digital representation of what the object actually looks like, making it easier to identify and analyze.

This technique could help speed up the training process for ultrasound technicians and other health care providers who use ultrasound. It could also be deployed for use in hospitals, for tasks such as using ultrasound to place a needle in the right location for a biopsy.

For training, this could make ultrasound more intuitive and more understandable. On the clinical side, it could be less time-consuming, more accurate, and also give health care providers more peace of mind. They wouldn't have to wonder if they missed anything."

Canan Dagdeviren, associate professor of media arts and sciences at MIT and senior author of the study

MIT graduate students Jason Hou and Shrihari Viswanath are the lead authors of the paper, which appears today in Nature Communications Engineering. Other authors of the paper include Bowen Wu '24 and two MIT Summer Research Program students, Cinay Dilibal, a senior at Dartmouth College, and Tanisha Shende, a senior at Oberlin College.

3D representations

Ultrasound imaging works by bouncing high-frequency sound waves off tissues in the body, which are then reflected back to an ultrasound transducer. The transducer converts these sound waves to electrical signals, which are used to create a 2D image of the tissue. Ultrasound technicians are trained to convert these images into a 3D mental representation of the tissue.

"It's a difficult skill to master, and there are long learning curves," says Hou. "The hardest thing is this mental tomography bottleneck where you're trained to reconstruct the 2D slices in your 3D mental space. That is a cognitive burden that can lead to inaccuracies in scanning."

To reduce that cognitive load, the MIT team thought it could be helpful to combine two technologies: 3D ultrasound imaging and augmented reality (AR).

Three-dimensional ultrasound imaging is occasionally used in fields such as fetal imaging and echocardiography, which is used to image the heart, but most 3D ultrasound imaging systems are expensive and not widely available. For this study, the MIT team used a real-time 3D system they developed recently for use in breast-cancer detection.

Their new system includes an ultrasound probe, slightly smaller than a deck of cards, that transmits information using a chirped data acquisition system (cDAQ). The probe contains an ultrasound array arranged in the shape of an empty square, a configuration that allows the array to take 3D images of the tissue below.

Because this system has fewer ultrasound elements than a typical 3D ultrasound system, it requires less power and is less expensive to build.

The data collected by the ultrasound probe can then be compressed and streamed into a 3D computer graphics engine called Unreal Engine, which converts the voxel data from the ultrasound image into a direct 3D representation of the object, with no loss of information. Wearing an AR/VR headset, the user can see this 3D rendering representing the internal structure, superimposed over the object's actual location - like X-ray vision. By tilting their head or approaching from a different direction, the user can see different views of the object, making it easier to identify.

Easier to use

The researchers tested their new technology, which they call AR-VIU (augmented real-time volumetric imaging in ultrasound), with a group of 18 participants. Nine of the subjects were experts in ultrasound technology (including sonographers and physicians), and nine had never used ultrasound before.

Each user performed identification tasks using four different ultrasound technologies. In one condition, they viewed 2D images on a regular screen, which is the way that most ultrasounds are now performed. They also viewed 3D images on a regular screen, as well as two augmented reality conditions: one 2D and one 3D (AR-VIU).

In one round of experiments, users were asked to identify an object embedded in gelatin - such as a spring, a ball, or a screw - inside an opaque container that was scanned with ultrasound. In a second set, they were asked to use a pen to mark the location of "tissue phantom" - a gel-like material engineered to mimic human tissue. This simulates the task of locating the right spot for a needle during a biopsy.

The researchers found that the AR-VIU system significantly improved all users' ability to identify and locate objects. The effect was especially strong for novices, who performed nearly as well as experts when using AR-VIU. When using the traditional 2D imaging system, experts performed much better than novices.

"Overlaying images with the anatomy and providing 3D visual context makes ultrasound significantly easier for novices to understand," Viswanath says.

In interviews after the experiments, most of the novices reported that they preferred the AR-VIU approach, with many saying that it made the tasks easier.

"The 3D system imposes less brain drain, it's more intuitive, and it's easier to understand what is happening in the targeted region," Dagdeviren says.

Many of the experts said they preferred the traditional 2D imaging because that is what they were accustomed to and had been trained to use. However, those experts also said they could see the benefits of the AR-VIU system in some situations, such as placing a needle for a biopsy or visualizing the movement of the heart wall during echocardiography.

The researchers are now working on further improving the resolution of the imaging and doing additional tests to demonstrate the accuracy of the AR-VIU technology.

The research was funded by the MIT Media Lab Consortium, the National Science Foundation, an MIT HEALS graduate fellowship, and an MIT-Tata graduate fellowship.