Augmented Reality for the Lab

“I have always been intrigued to find out if there was a better way to practice pathology,” says Liron Pantanowitz, Director and A. James French Professor of Anatomic Pathology at the University of Michigan. “Most recently, I have been extremely impressed with the capability of augmented reality (AR) microscopy.” Having tested the technology, he reports, “It was awesome and delivered just as expected, bringing AR right to my microscope. There are so many use cases for augmented and virtual reality in pathology – I am just getting started.”

Swati Satturwar, Genitourinary Pathology Fellow at the University of Pittsburgh Medical Center, says that digital pathology was limited to tumor board presentations during her residency – but when she took up her cytopathology fellowship position, she began using it on a daily basis. Eventually, she took on a research project using AR microscopy (ARM) with real-time image analysis for Ki-67 quantification in cell block material of neuroendocrine tumors. The project involved comparing conventional manual counting methods to novel AI-based methods using digital image analysis and ARM. “I am excited to work on more projects exploring the potential of these technologies to complement pathologists,” Satturwar says. “I am also excited that UPMC is going to adopt Ibex AI algorithms for prostate biopsies. As a GU fellow, I will have the opportunity to be a part of this adoption of novel, AI-based diagnostics in routine clinical practice.”

Powerful promise
Of course, AR and VR aren’t new technologies. “I’m not much of a video gaming person,” says Pantanowitz, “but I tried AR/VR briefly for gaming and had some fun. Of course, my kids love it. At one stage, I couldn’t get them to stop playing!” Augmented reality has also taken him to museums and far-flung locations such as Machu Picchu. “Most recently, I nearly bought a house after just taking a 3D virtual tour!” he says. “It was almost as good as a real walkthrough of the property.” 

Satturwar prefers to use the technology for social purposes. “I use AR while taking fun photos with my smartphone – for instance, to add a variety of stickers,” she says. VR is reserved for video games, from bowling to roller coasters.

In the lab, though, the tech takes on a more serious tone. “I am a huge proponent of digital pathology,” explains Pantanowitz. “What I love most about it is that, once a slide is digitized, it opens up a world of possibilities. The applications with a digital image are endless.” The biggest obstacle to pathology’s digital transition has been the need to acquire the image first. Photographing slides is cumbersome and doesn’t give you access to the entire slide, whereas scanning the glass slide is time-consuming and requires expensive hardware and software. “With ARM this inertia is removed. All you need to do is attach an AR device to your microscope and voila – you are ready to go. That’s what inspired me.”

Originally, Satturwar didn’t know ARM technology existed. “In the first month of my cytopathology fellowship, Liron Pantanowitz asked me if I would like to participate in a research project using ARM. He showed me a video of the technology. I was amazed by its potential and said yes right away.” 

The ARM Satturwar uses is an Olympus microscope with an Augmentiqs AR device attached between the microscope’s objectives and eyepiece unit and an inbuilt camera to capture high-quality images. The images can be viewed through the microscope’s binocular lens or displayed on the monitor of an attached computer. It can overlay additional information (whether computer-generated data or the pathologist’s manual annotations) onto the original microscopic field of view (FOV) in real time, without having to first digitize a glass slide. 

ARM even enables real-time image analysis on the glass slides. How? By integrating AI algorithms to generate a composite FOV that can be used for advanced data collection without altering the traditional manual pathology workflow or the optical quality of the microscope. This modified “smart microscope” can be used for a variety of diagnostic purposes, including:

• simple measurements (e.g., size and depth of tumor or lymph node metastasis)
• immunohistochemical stain quantification (e.g., Ki-67 proliferation index)
• diagnosing non-neoplastic diseases (e.g., myopathy and non-alcoholic steatohepatitis) 
• cancer diagnosis (by integrating AI such as deep machine learning algorithms)

The high-quality digital images the microscope produces can be used for telepathology, tumor board presentations, frozen section peer reviews, teaching, and research.

Pros and cons
Augmented reality – and ARM in particular – has a lot to offer in the lab. Its benefits include:

• AR devices can be attached to any conventional light microscope to convert it into a “smart microscope.”
• Real-time image analysis on glass slides by avoiding the time-consuming process of digitizing glass slides prior to image analysis. This decreases the disruption to routine workflow in a busy pathology practice.
• Minimal technical skills required to operate ARM, unlike whole-slide scanners that require special technical expertise.
• More affordable than a conventional whole-slide scanner.
• Not associated with simulator sickness, which is known to occur with wearable AR/VR devices.

But though AR and VR look like promising tools for pathology, the technology has a long way to go before it’s ready for routine clinical use. Some existing devices lack sufficient image resolution for medical-grade work; others carry privacy concerns when sharing protected health information over the Internet. “Of course, the instant you turn a device into a ‘medical instrument,’ everything suddenly becomes more complicated,” says Pantanowitz, citing challenges from liability to regulations to vendor support.

Satturwar agrees. On her list of things that need to be addressed before integrating ARM into routine pathology workflow are large-scale studies validating its uses; increased awareness of ARM among trainees, practicing pathologists, and researchers; and the development of affordable, yet sophisticated, deep learning algorithms for pathology diagnostics.

But which is better – AR or VR? Is it preferable to combine reality with computer-generated information and superimpose a digital image onto a view of the real world or to provide a completely computer-generated environment that merely simulates a real experience?

Pantanowitz sees benefits and shortcomings to both. “VR is definitely more immersive – but, for pathology work, it’s just not practical and can cause issues such as motion sickness,” he says. “AR, on the other hand, is more practical, additive, and offers more possibilities (1).” Expanding on AR’s user-friendliness, Satturwar adds, “ARM is more likely to be adopted by users because it integrates into the existing microscope and does not require any wearable accessories.” AR technology offers the potential of greater accuracy, efficiency, and reproducibility for simple morphometric measurements or stain quantification. And, coupled with AI algorithms, it can enhance lab workflow and decrease turnaround time by automating multiple diagnostic steps.

The learning curve
“AR is ideally suited for education,” says Pantanowitz. “It offers learning that is more engaging, fun, helps explain abstract concepts, and can reach more people. This is becoming big business.”

Because AR devices can be attached to multi-headed microscopes, educators can use ARM to annotate important pathology features, such as mitotic figures. Some devices even have a stage-tracking facility that students can use to follow the educator’s exact movements in reviewing the slide – helping them learn to navigate difficult cases. Satturwar says, “ARM can save time and improve trainees’ educational experiences by integrating different features, such as more accurate measurements and automated stain quantification.” Manual counting of Ki-67 or H-score for breast biomarkers is time-consuming and shows interobserver variability – whereas ARM can make the process faster and more accurate so that trainees can focus their time on difficult cases.

Here again, AI can be a hero. Real-time integration of AI – for instance, deep learning algorithms – lets multiple users see the same FOV with superimposed heat maps for tasks such as cancer detection. Telepathology allows screen-sharing or even remote double-scoping – an especially helpful feature during a pandemic. “One of the main barriers to adoption of the newer technologies by experienced pathologists is a reluctance to try new things because it differs from what they learned during training,” says Satturwar. “If residents and fellows get this learning experience early on, it will increase adoption.”

AI’s leading role
“These two promising technologies are synergistic,” says Pantanowitz. “Combined, they deliver a win-win solution.” Pathologists with an AR/VR platform can essentially plug and play AI solutions designed to assist them – something Satturwar, Pantanowitz, and colleagues tried to great effect in a study on Ki-67 scoring in neuroendocrine tumors (2). Satturwar says, “The study allowed us to rapidly quantitate a Ki-67 index without prior digitization of glass slides and demonstrated near-perfect agreement with the printed image manual count method.” She also points out that newer, more robust deep learning algorithms with convolutional neural networks have outperformed pathologists in some studies using whole-slide imaging modalities (3) – and that the same may eventually be true of AR coupled with AI/deep learning algorithms.

AI is revolutionizing the way pathology is practiced today. Because of the COVID-19 pandemic, the Centers for Medicare & Medicaid Services (CMS) in the USA have approved remote sign-out using digital pathology. But that’s only the first step; pathologists need enough training and confidence in digital pathology to take advantage of it – and there can be not only technical, but also financial barriers to a full digital transition. “AR/VR tools will give us a platform to do what we already do every day, but better,” says Pantanowitz. “They can give more precise, accurate, and standardized results and scores with the help of simple image analysis or more sophisticated AI. They can even offer users the ability to work remotely – a big deal now due to the pandemic.”

 “ARM presents a cost-effective model that allows seamless integration of AI algorithms without digitizing the glass slides, thereby eliminating the time and cost of digitization,” says Satturwar. “AR also allows focus adjustment, which is essential during the review of unique cytology preparations.” The technology’s benefits are not limited to state-of-the-art labs, though; AR could eliminate disparities between remote and resource-poor health care facilities and their more affluent counterparts.

An augmented routine
How soon might we see AR in the lab? There are still a number of hurdles to clear before the technology can be implemented for routine use – including i) appropriately dealing with the issues typically needed to convert these instruments into “medical devices” (such as regulatory approval), ii) published validation studies to get pathologist buy-in, and iii) early adopters to actually deploy these systems and demonstrate their use cases, limitations, and return on investment.

“The use of AR in routine pathology workflow is possible in the near future,” says Satturwar, “after large-scale validation studies confirm its utility and technologies like ARM get FDA approval.” At the moment, few vendors offer ARM, which means that many laboratorians are unaware of its existence, let alone its promise. “We need widespread awareness to encourage practicing pathologists to adapt to newer, more accurate, and more efficient technologies. Vendors must also develop affordable AI algorithms for common diagnostic specimens according to laboratories’ needs.” Even after these things are in place, though, she cautions that labs must develop policies regarding the use of AR to enhance the existing workflow before bringing it into routine use.

What of regulatory concerns? At the moment, there are no standardized guidelines on validation or licensing requirements for AR/VR. “It’s one thing to do a research study to demonstrate clinical feasibility,” says Pantanowitz. “It’s a totally different ball game when you want to start using technology for routine patient care in clinical practice. Within this clinical context, there are regulations that apply to the manufacturer (such as FDA approval) and those that apply to the lab (such as compliance with accreditation needs or establishing a QA program).” Satturwar also recommends that labs conduct their own validation studies before fully adopting the technology. Nonetheless, her excitement remains untempered. “With the integration of AR coupled with AI into routine pathology, the future is bright.”

AR and VR technologies aren’t meant to replace traditional pathology – or the pathologist. Instead, they should be adopted as a complement or enhancement to manual pathology workflow. Satturwar warns that labs need not only a quality assurance system for all AR- or AI-assisted diagnostics, but also a full understanding of the tools’ limitations. For instance, AI algorithms are trained for diagnosing certain diseases – which means they may miss co-existing pathologies for which they are not trained. Of course, cost is always a concern. “Look for a vendor who can provide affordable AI according to the needs of a given laboratory,” she advises – a task that may become easier as appropriate billing codes for computer-aided diagnosis are established.

Pantanowitz adds, “There is always the risk of trying out new technology just because it is novel. I would advise pathologists to avoid the ‘shiny object syndrome.’ However, if there is a good niche application (for instance, to enhance training/education, perform better path-rad correlation, or provide a platform for easy image analysis) that justifies use of the technology, then go for it!”

But despite these cautions, Satturwar and Pantanowitz believe the clinical laboratory is ready for augmentation – perhaps even before entertaining thoughts of a full digital transition. “Although converting to complete digital workflow is optimal, there are certain challenges,” says Satturwar. “AR technology that offers similar applications without prior digitization of slides is a more affordable and cost-effective route to adopting novel technologies into routine pathology workflow.” 

The final word? Pantanowitz sums it up in an apt paraphrasing of Darwinian theory: “It is not the strongest of the species that survives, nor the most intelligent. It is the one that is most adaptable to change.”