Pathologist-Centric AI

Most people have some understanding of artificial intelligence (AI) – but what exactly does it mean in the context of pathology? A pathology AI system is a computer program that assists pathologists in their work or provides automated pathology. The key capability of such a system is to analyze digital slide images using image analysis and “machine learning” – another buzzword! Machine learning is the process of learning a task (for instance, providing a diagnosis or a score) or a sub-task (such as classifying cells into different cell types) from data. There are many approaches in machine learning, including decision trees, random forests, and deep learning. You may have heard of some of these – in particular, the latter.

Deep learning

In recent years, we’ve seen a great deal of hype around deep learning. It’s a process that has overcome major challenges in computer vision, allowing us to implement feature detection successfully where image analysis algorithms failed. A deep learning network can learn highly complex visual features from image data, achieving performance that may even equal that of a human expert. Deep learning requires huge amounts of data and significant processing resources. But recently, with the increase of processing power and, in particular, the use of GPUs, it is now possible to train deep learning networks successfully.

The first deep learning network to achieve a major breakthrough was AlexNet, which in 2012 significantly outperformed all previous approaches on the ImageNet challenge (a large visual database designed for object recognition software research). Since then, more efficient, higher-performing systems have been introduced – and, because pathology is a visual task, it is more than understandable that deep learning is now coming to pathology as well. In 2016 and 2017, Grand Challenges in Biomedical Image Analysis (CAMELYON 16 and CAMELYON 17) were run to encourage the development of programs to detect cancer metastasis in lymph nodes. Both challenges were clearly dominated – and won – by deep learning. These are complex programs, which explains both their high performance and the difficulty of designing them. It’s no longer just about finding the right hyper-parameters; now, creators must design new network topologies. It’s not just a science, but also an art!

Because of this complexity, many applications begin by reusing existing designs that have been proven in other applications. For example, the winner of CAMELYON 16 reused and modified GoogleNet, which won the 2014 ImageNet challenge. But because pathology applications are different from general-purpose image recognition tasks, laboratory professionals could gain a lot by handcrafting an appropriate net topology for pathology applications – especially focusing on cell data.

Decision trees

It’s only in recent years that academia has started to use digital slide images, but today, academicians are driven by the deep learning hype. It has great potential in situations where feature detection presents a challenge for traditional image analysis, but it comes at a price. The data sets used to train deep learning algorithms are expensive, there’s always the risk of bias from the training data, and there is no transparency in the decision-making process. Why did the machine classify one group of cells as benign and another as malignant? With deep learning, it can be hard to know.

Decision trees are like hierarchical flowcharts – a structure humans tend to find both intuitive and transparent. Their advantages include:

• no data normalization,
• proper handling of missing data,
• heterogeneous data (numerical, ordered, and categorical),
• intrinsic feature selection,
• multiclass,
• multioutput, and
• fast predictions.

Decision trees sound perfect for use in medical devices – but can they deal with the complexity of analyzing digital slides? A natural extension that provides a more complex machine learning approach uses “random forests.” These share the same advantages as basic decision trees; they do lack transparency in the decision process, but the important features can still be identified. Before deep learning took the world by storm, random forests had success winning competitions used to evaluate different machine learning approaches.

The key problem for any pathology AI system is the variation between different patient types. In a disease state, no two patient samples look identical. To distinguish between different cell types – a task any machine learning system has to accomplish – we must observe that the same cell type can have different, and sometimes even contradictory, characteristics in different patients.

Figure 1 shows two patients that illustrate the concept of contradictory characteristics: patient A’s stroma cells have the same characteristics as patient B’s tumor cells. If we create a machine learning system trained only on patients with the same cell characteristics as patient A, it will fail when it encounters a patient like B, illustrating the potential bias that can originate from the data used for training. To ensure that the machine learning system would be able to learn the characteristics of all patient types properly, we need a lot of training data. Getting a machine learning system from 90 percent performance to 95 or even 99 percent becomes exponentially harder as remaining exceptional cases are hard to come by.

Now, when we create a machine learning system trained on patients belonging to a variety of patient types, the system would have to learn somewhat contradictory data. We would need to use a complex machine learning approach that could learn highly complex visual features with different contexts. Obviously, deep learning would be the right tool for that job – but its lack of transparency will eventually lead to legal and regulatory hurdles as pathologists and their AI partners make medical decisions that put human lives at risk.

A new diagnostic AI

In pathology, there is no critical need to use machine learning to learn the visual features in a histology slide. We are not looking at arbitrary objects in an uncontrolled environment; we are looking for cells that a) are of a certain size, b) have three cell compartments – nucleus, cytoplasm and membrane – and c) can only be stained by a small number of stains with distinct colors.

Traditional image analysis does a good job of detecting cells and measuring the wealth of information on a histology slide. But what do we do with that information once we have detected the cells and logged all of the biology-motivated information? A machine learning system that uses multiple “patient type”-specific classifiers and is based on cells, not pixels, requires no training data, yields excellent performance and provides transparency into the decision process! Here is how it works:

1. When we encounter a patient that belongs to a new type, we create a new “patient type”-specific classifier. A pathologist, using their expertise, identifies a few example regions for the different cell types (for instance, tumor and stroma) and trains a new classifier “on the fly” that is then used to classify all of the cells on the slide. Proper controls are implemented by having the pathologist verify proper classification of the cells. New example regions are added and the classifier retrained until the pathologist is satisfied with the cell classification.
2. With any new patient, we first select the best classifier from all existing “patient type”-specific classifiers and use it to classify all of the cells on the slide. A simple, robust method that nicely illustrates the selection of the best classifier is to have a pathologist identify an example region for one or more cell types and select the classifier that provides the best performance on those regions. When the pathologist verifies the classification of the cells, they may decide that it is not good enough, which means that the new patient belongs to a new patient type and a new classifier needs to be created (go to step 1).

It’s clear that, if we were to create different classifiers for different patient types, a simple decision tree using just a single feature with a single threshold would provide excellent performance and easily interpretable decisions. The results obtained by machine learning match nicely with what we see by eye (the separation between tumor and stroma cells based on nuclei size in patient A and on density of cells in patient B). Limiting machine learning to a specific patient type and using cell data simplifies the problem considerably. It can yield excellent performance with simple approaches like decision trees, which consist of easy-to-understand hierarchical flowcharts and only require data from a few regions for training. The training is ultra-fast and can be done “on the fly” in an interactive, iterative workflow. A decision tree based on biology-motivated features provides easily interpretable data and a meaningful grouping of patients by type.

A pathologist-centric system

A pathologist-centric AI system (see Figure 2) semantically segregates tissue analysis into three distinct parts: a) cell detection, b) cell classification, and c) measurements that provide rich information. It’s at the junction between classification and rich data that pathologists provide their expertise and the proper controls for the system in a natural way.

Traditional image analysis is good at detecting cells because the imaging process is very controlled and the image contains only tissue prepared using a controlled process. Computer vision is an area where deep learning has achieved major breakthroughs over the last few years, so deep learning algorithms can be used for meaningful cell detection. And, because pathologists have been looking at cells on histology slides for over a century to make their assessments, a cell-based representation of AI data allows pathologists to understand and interact with the system intuitively. Markup images of the cells on a slide contain all of the available histological information – making them the perfect abstraction layer for a pathology AI system.

Cell classification is a critical step that any pathology AI system needs to accomplish; it ultimately determines the complexity of the machine learning approach and our ability to understand the decision-making process. As noted, we propose a “patient type” machine learning approach based on cell data, wherein pathologists contribute their expertise by using example regions for the different cell types to train patient type classifiers. This approach provides full transparency into the decision-making process. Of course, the pathologists must also serve as controls to verify cell detection and classification!

With those elements properly verified, the measurements that provide rich information for tissue can then be viewed as simple computer algorithms. This is the final step, but no less key than those that go before it. After all, this is the data we use to develop specific diagnostics, companion diagnostics, and prognostics for our patients – and we need these things to enable big data for pathology. It’s our hope that, in the near future, pathologists and AI systems will work together for better data management and patient care!