Toward Higher Standards in Viral Diagnostics

• When it comes to diagnostics and treatments, standardization is key – but often inadequate
• Without the ability to compare between assays, it’s difficult to determine whether differences in the results of two samples are meaningful 
• Adequate assay standardization requires reliable units and trustworthy reference materials
• As technology – especially next generation sequencing – advances, we need to develop new reference materials to ensure we can keep evaluating test reliability

How much insulin is present in a preparation of pig pancreas for treating diabetes? How much bacterial antitoxin is present in the serum of a horse immunized against infection? These are just two of the questions that scientists were grappling with 100 years ago and ultimately gave rise to the National Institute for Biological Standards and Control (NIBSC) – an organization that has spent the last half-century tackling the challenge of measuring biological medicines. Although such medicines may seem crude to the modern practitioner (pig pancreas or horse serum, anyone?), the methodologies we developed then still have a lot to teach us, even in the age of synthetics and biosimilars.

In the 1980s, we recognized that the lessons we’d learnt from classical measuring challenges could also be applied to the difficulties faced by the blood industry with the emergence of blood-borne viruses like HIV and hepatitis C. Molecular amplification assays had just been invented, and we immediately recognized that they gave us the power to improve virus detection. But it wasn’t as clear-cut as it sounds. When using nucleic acid amplification techniques (NAATs), we all believed that we could detect a single molecule of target sequence. But when we compared two or three results, everybody’s molecule was slightly different. So the big question: was anyone correct? NIBSC’s offer to the virology community was not that we were experts in NAAT assays, but that we could make stable materials that contained a consistent amount of target sequence – standards that labs could use for comparison. One thing we wanted to target was the concept of a “copy.” What is a copy? Patients, clinicians and researchers believe they know, but, in practice, each assay measures it slightly differently. We were very keen to switch to a unit that would allow us to standardize between assays. Working with the WHO expert Committee for Biological Standardization (ECBS), we began developing reference materials that allowed the amount of target to be calculated as a relative potency in arbitrary, but defined International Units (IUs). We rolled our new standards out in the blood-borne virus industry in the 1990s and, since then, we’ve seen significant improvements to the quality, sensitivity and comparability of different diagnostic assays and platforms. Our next task is to extend what we’ve learned to a broader range of clinical viral diagnostic assays.

Setting standards

One of clinical virology’s challenges is that, because of the mobility of our patient population and the consolidation of pathology services, we have to ensure that differences between assay results are caused by actual changes in the amount of target pathogen, rather than technical differences in measurement. Disentangling those two aspects is difficult – but vital. NIBSC has been working with the clinical virology community to develop a series of international standards. We want stable preparations of clinical targets that can last for years, and we want to be able to produce thousands of vials that all contain the same material in the same amounts. If different labs measure different contents, we need to know that the problem lies with the equipment or method, rather than the analyte itself. You may be thinking, “If each lab measures slightly differently, how can we be sure of what’s in the tube in the first place?” The answer is that we actually rely on international collaborative studies – in other words, we let international expert laboratories tell us how much there is in our vials and we establish a consensus through this collaborative study process.

Making a reference material

The starting point for a new NAAT standard is a meeting called SoGAT – “Standardization of Gene Amplification Techniques.” We bring together expert stakeholders from a variety of international sources and identify our main priorities. Which tests are most important for the international community? Over the last decade, for instance, it has been the herpes viruses, which affect patient health and clinical management after organ transplantation. The stakeholders tell us what they need, we bring the proposal to the ECBS for endorsement, and then we work to produce and validate a candidate reference material. We also need to ensure that our calibration references act exactly like clinical samples – an attribute known as commutability. The volume of material required to make an international standard is too great to rely on clinical samples alone. As a result, we need to show that the standard works like a clinical sample in every type of assay in which it is used (see Commutability Conundrum). To that end, we must include a number of clinical samples as part of the collaborative study. This used to be straightforward when working only on blood-borne viruses, but now that we’re dealing with whole blood, urine, cerebrospinal fluid, and so on, it’s much more complex and time-consuming. Ultimately, though, once we’ve completed our studies and analyzed the data, we submit a report to the ECBS – and that, hopefully, results in a new international standard. On average, the process takes about two years – but when we’re dealing with an emerging virus like Ebola or Zika, we try to find ways of shortcutting the process; for instance, by reducing the number of laboratories that evaluate our standard or by developing novel evaluation or treatment approaches (1). We managed to produce an international standard for Ebola virus within nine months, which gave doctors an extra year or more of reliable clinical diagnostics.

Standardization, quality assurance and validation aren’t the same thing – but unfortunately, they’re sometimes used interchangeably. Standardization is a metrological term for the ability to compare assays performed through space and time. That might mean assays conducted in the same laboratory at different times, or assays conducted at the same time in different laboratories. The goal of standardization is to allow us to compare the outcomes of those different assays. Quality assurance and validation have more to do with individual laboratories – how “good” their assays are. Those terms deal with things like reproducibility and detection. Can we repeat the same assay on the same sample and get the same result? Can the assay reliably detect its target in a wide range of clinical samples?

Weaknesses to watch

Molecular diagnostic assays provide a level of specificity and sensitivity that was not previously available, and have the power to improve clinical management of patients. But that same specificity and sensitivity is also a potential weakness; subtle changes in pathogen sequence can lead to drastic differences in measurement. Suddenly, a pathogen we could readily measure might be completely missed! It’s vital for diagnostic professionals to be aware of that significant Achilles heel. An even greater concern is the “copy” concept, which we’ve already established doesn’t always mean quite the same thing. It’s not a magic number – it carries an inherent uncertainty that clinicians and patients sometimes overlook. In clinical chemistry, assay methods are very robust and the resulting numbers are reasonably accurate; the challenge with current molecular diagnostic techniques is that there is much greater potential 
for variability and this can be critical if guidelines state specific clinical actions should occur at a specific level. We recently sent identical vials of polyomavirus to different laboratories for quantification as part of a study. The great news is that each laboratory’s results were highly reproducible – meaning that they had good quality assurance systems in place. The bad news is that there were massive (1000 fold) differences in the amount of virus each laboratory detected (2). Such disparity can significantly affect clinical management, and that’s why we need to embrace the process of standardization. Quality assurance alone isn’t enough; we also need reliable inter-laboratory comparison. The good news is that most of these differences disappear when they are calculated as relative potencies (2).

Many users see plasmids as good reference materials. The problem is that they’re fine for the amplification step, but they don’t look like clinical samples. We demonstrated that very clearly when we established the international standard for cytomegalovirus; we had both a plasmid and a virus that was identical in sequence, and only the virus itself covered all of the steps in the process (extraction, amplification, and so on). If your standard doesn’t incorporate every step of the process, it’s highly unlikely to improve the quality of the assay. It all comes down to commutability – making your reference resemble a clinical sample as closely as possible.

What comes next?

Beyond virology, molecular diagnostics are increasingly being applied to all types of infectious agents. The artificial distinctions between virologists, parasitologists and bacteriologists are being broken down at the clinical diagnostic level for infectious diseases. Instead of clinging to those distinctions, we should learn from one another by sharing new ideas, technologies and reference materials.

Point-of-care testing is another field that’s on the rise, and it carries its own set of standardization challenges. These are limited numbers of small samples that can be processed in machines designed for use in the physician’s office. But they tend to be all-in-one packages; the technology is built into the machines by the manufacturers, who purportedly address quality assurance and validation – but not necessarily standardization. It’s quite difficult to include appropriate reference materials when a machine may only be able to process one or two clinical samples at a time.

A final trend, which may supersede all of the current technologies being applied to diagnostic NAAT assays, is the rise of next generation sequencing (NGS). Such technology overcomes a key pitfall – namely, the fact that a minor change in a target pathogen’s sequence may cause current molecular assays to miss it completely. However, it is important to note that NGS-based detection of infectious agents is no less reliant on standardization. One recent NIBSC publication showed that when we put similar amounts of 25 viruses into a tube and asked laboratories how many viruses they could detect with NGS, they each came up with very different numbers – and, not only that, the amounts they detected were also highly variable (3). In fact, some of the labs even discovered viruses we hadn’t put into the vials in the first place! It just goes to show that NGS, while very powerful, is still a long way from being standardized. It’s a problem we need to solve quickly, before the technology gets too much bad press for “failing to deliver on its promises.”

We don’t have the answers to all of these challenges yet, but at least we’ve begun to identify the scale of the problem. We’ve come a long way from pig pancreas and horse serum days – but as technology advances, we must tackle new issues as they arise to ensure that our ability to diagnose infections keeps pace.

Newsletters

Receive the latest analytical scientist news, personalities, education, and career development – weekly to your inbox.

I have read and understand the Privacy Notice*

Subscribe

References

1. G Mattiuzzo et al., “Development of lentivirus-based reference materials for Ebola virus nucleic acid amplification technology-based assays”, PLoS One, 10, e0142751 (2015). PMID: 26562415. S Govind et al., “Collaborative study to establish the 1st WHO International Standard for BKV DNA for nucleic acid amplification technique (NAT)-based assays”, WHO, Accessed February 14, 2017 http://bit.ly/2kPlfGo ET Mee et al., “Development of a candidate reference material for adventitious virus detection in vaccine and biologicals manufacturing by deep sequencing”, Vaccine, 34, 2035–2043 (2016). PMID: 26709640.