Deep Learning for Molecular Tumor Biomarker Prediction

When choosing a treatment to target an individual tumor’s specific weaknesses, its molecular properties are critical. And yet, despite the importance of this information, our most accurate methods of assessing molecular properties are expensive and often not routinely performed. We need a better solution – and artificial intelligence (AI) may offer one. Using AI to examine microscopic images of tumor tissue could provide a more cost-effective way of identifying individual tumors’ molecular characteristics to inform treatment decisions.

Histopathology, of course, is the gold standard for diagnosing cancer. A pathologist, examining a prepared tumor sample slide under a microscope, decides based on appearance whether or not cancer cells are present. Although pathologists are experts in diagnosing cancer, they can still assess only limited tumor properties, even with the aid of microscopes and special stains. At the University of North Carolina, we investigated whether a computer could find features to predict molecular biomarkers – features too complex for pathologists to assess visually. Our answer, with a focus on breast cancer, was yes – and other researchers have recently found that the same is true for other cancer types.

Molecular subtypes for tumor stratification

One goal of cancer research is to identify cancer subtypes based on the characteristics of tumor cells. A better understanding of these subtypes can help identify both causes of and potential treatments for the disease. They also allow health care teams to personalize treatment to the needs of each patient by selecting the best approach for the disease subtype and prognosis.

Recently, we’ve seen great success stratifying tumors into subtypes using their molecular properties. For example, breast cancer is divided into five different subtypes that add prognostic information and treatment efficacy insights (1). Normal-like and luminal A and B cancers are hormone receptor-positive and generally have the best prognosis. HER2-enriched ones are often successfully treated with therapies aimed at the HER2 protein. Basal-like cancers tend to be more aggressive and have a poorer prognosis. However, the technologies we use to assess these molecular properties are expensive and the analyses are time-consuming. Most are not routinely performed – meaning that some patients who could benefit don’t receive the testing – and labs with limited resources may not perform them at all. To complicate things further, tissue is often a limiting factor; in many cases, only a small amount of tissue can be excised from a tumor, leaving none for additional analyses beyond microscopic examination.

New studies keep identifying more molecular properties of potential clinical value, each requiring its own tissue sample and processing procedure – but current workflows are not designed to incorporate this many tests. Although comprehensive molecular testing will be difficult to implement at scale, tissue staining is common practice and imaging of such samples will become increasingly available as more laboratories transition to digital pathology.

Histological imaging for diagnosis

Hematoxylin and eosin (H&E) slides are inexpensive to produce and have a short turnaround time, which is why it is so well-established and widely used to diagnose cancer, assess its aggressiveness, and study tumor tissue. However, pathologists cannot use H&E staining to assess the molecular biomarkers that guide treatment. Image analysis methods can provide a screening mechanism to identify patients who would benefit from further molecular testing or act as an alternative when molecular analysis is not possible.

Right now, we can use H&E to look at the size and shape of cell nuclei, their arrangement, the presence of other tissue structures (such as glands), and the presence of specific cells and cell structures (such as lymphocytes or mitoses). But, to go beyond the complexity of what human experts can assess visually, we must rely on new image features – and we must learn those features from the images themselves. This is known as feature or representation learning. The best-known (and currently most powerful) tool in this set is deep learning.

The rise of deep learning in histopathology

A computer can learn patterns in images so that it can make predictions based on those patterns – this is the essence of deep learning. After training on a data set of images and labels (such as biomarker status), the model can predict these labels on new, never-before-seen data. The model consists of multiple layers of features, with higher-level concepts built upon the lower-level ones. Going up the hierarchy, the features increase in both scale and complexity. Similar to human visual processing, the low levels detect small structures such as edges; intermediate layers capture increasingly complex properties like texture and shape; and the top layers of the network can represent objects and complex properties like tissue architecture.

This method has previously shown success for finding mitoses, segmenting tissue types, and detecting tissue structures. However, those features all have one thing in common: pathologists can see them under the microscope. The next step forward is to apply this powerful technique to higher-level (even tumor-level) properties for which a pathologist cannot provide detailed annotations.

Deep learning performs very well when given a lot of training data – often millions of examples. However, it struggles when (as is usually the case with medical images) training data is limited. A common solution for small datasets is “transfer learning”: a model is trained on one data set, usually a large benchmark set of photographs called ImageNet, and then applied to a different one. In my work with breast cancer, I used the pre-trained model to compute features on H&E images and then trained a classifier to predict the biomarkers. Subsequently, I fine-tuned the model for improved performance on breast cancer.

Before working on breast cancer, my first experience with image analysis for pathology was in trying to predict melanoma mutations from H&E images. We used handcrafted features – the size, shape, and texture of nuclei and their spatial arrangement – properties that a pathologist looks for when diagnosing and grading cancer. Ultimately, however, we were not able to distinguish tissue samples with the mutations from those without.

This was in 2012, when deep learning was in its infancy. The research that popularized deep learning was published that same year (2), and open-source toolkits started to become available over the next few years. It was not until 2015 that today’s more easily accessible libraries were first released. These developments in machine learning laid the groundwork for many pathology applications, including biomarker prediction from H&E.

Molecular biomarker prediction with deep learning

Recent work at New York University has shown that the exact mutations we looked at in melanoma are predictable from H&E images – using deep learning (3). Similar results have been found for point mutations in breast, colorectal, gastric, lung, and prostate cancer. A team at the Cleveland Clinic found that tumor mutational burden – a measurement of mutations carried by tumor cells – is predictable for bladder cancer (4). Detecting mutations is typically done with DNA sequencing, which is costly and time-consuming, but can guide targeted therapies.

Our work at the University of North Carolina addressed genomic subtypes of breast cancer. Gastric, lung, and colorectal genomic subtypes have also seen success at other institutions. Genomic subtypes stratify patients based on the activity level of specific genes, providing prognostic information and helping to guide treatment decisions. Although assays are available for some genomic subtypes (5), RNA sequencing is still the most accurate method for distinguishing between the subtypes.

Going beyond mutations and genomic subtypes, we used deep learning technology to predict estrogen receptor status on breast cancer. Other researchers have since tested a simpler method on a larger data set and further expanded to include an additional 18 protein biomarkers (6). The standard method for assessing protein biomarkers is immunohistochemistry – an alternative tissue staining methodology that is time-consuming, costly, and requires a pathologist’s subjective interpretation.

Even the presence of viruses can be detected from H&E images. Human papillomavirus was identified in head and neck cancer and Epstein-Barr virus in gastric cancer (7). Both viruses are major causes of human cancer and knowing about their presence impacts treatment decisions. Molecular tests for these viruses are expensive and not available everywhere – which leaves many laboratories reliant on staining to confirm their presence.

Deep learning creates new possibilities

The first step in each of these approaches is to identify the tumor regions in each tissue slide. Some groups rely on pathologists to delineate the tumor; others automate this step of distinguishing tumor from non-tumor tissue. All groups rely upon deep learning’s abstract features to predict the molecular biomarkers. Traditional handcrafted features are just not powerful enough for these tasks. It is only through more complex properties – visible to machines, but beyond the capability of even the most expert pathologists – that we can now screen for some molecular biomarkers from H&E alone.

Deep learning is opening up a new world of possibilities in capturing properties that are too complex for human pathologists. It is a possible screening opportunity for any marker that can guide further testing and treatment selection. Although pathologists are – and will always remain – the experts in their craft, these additional insights can assist in targeting therapies for each patient.