The need for reproducible and standardized metrics for SARS-CoV-2: What we learned from COVID-19

The methods used to quantify nucleic acids are similar across different applications, whether it’s measuring SARS-CoV-2 viral loads or examining cancer variants, or even performing a non-invasive pregnancy test. While there are subtle differences in what is being measured, fundamentally, the process involves reading and quantifying RNA or DNA sequences.

However, when it came to exploring quantification of SARS-CoV-2 during the pandemic, knowledge established over a decade for the quantification of viruses like HIV were surprisingly not used, leaving some virologists and molecular biologists wondering why established concepts were overlooked. In this SelectScience article, we speak with Prof. Jim Huggett, a science fellow at the UK National Measurement Laboratory, LGC, and visiting professor at the University of Surrey, who performed a study to examine whether the metrics used to diagnose and stratify COVID-19 patients were indeed accurate.

What counts as a positive or negative COVID-19 diagnosis?

When the coronavirus pandemic began to surge in April 2020, there was an urgent need to diagnose patients and increase the testing capacity while upholding high standards. “At an open government meeting, I asked how quality would be ensured in this process,” recalls Huggett, an expert in measurement science. “I was told that because there are millions of SARS-CoV-2 viral copies in a patient sample, the PCR test would easily catch the result, as a simple yes/no answer was enough to determine whether the patient was infectious.”

Huggett, who has a keen interest in setting measurement standards for molecular diagnostics and hopes to make metrics more robust, accurate, and reproducible, watched closely as COVID-19 diagnostic testing unfolded across the world. In no time, RT-qPCR became the gold standard method for SARS-CoV-2 measurement. A positive RT-qPCR is reported as a Cq or quantification cycle (also termed cycle threshold or Ct). This would indicate the presence of RNA but has been proposed as a surrogate for the viral load carried by a patient, potentially to determine their risk of spreading infection. Furthermore, the World Health Organization (WHO) has used Cq cutoffs to advise diagnostic manufacturers to set targets for viral load, with point of care tests needing to detect a Cq value of at least 25.

“While all these were valid responses to an ongoing pandemic, I wondered how there was so much confidence in using Cq as a unit for a new virus that’s only been around for a few months, especially when these metrics were proposed to stratify patients,” notes Huggett.

To investigate further, Huggett initiated a collaboration with three COVID-19 testing laboratories across the UK, Belgium, and Korea to explore the quantitative ranges of patient reports and analyze just how standardized the diagnostic processes were. The results of this study were published in Clinical Chemistry in 2021¹.

Why Cq cutoffs aren’t the best approach

The group analyzed Cq distributions of SARS-CoV-2 results from 6000+ patients across three different clinical laboratories and found a 1000-fold difference in results for the same Cq value. RT-qPCR Cq values have generally been known to fluctuate across laboratories based on technical and experimental variables, but in the context of patient stratification, this 1000-fold difference can have serious consequences. When the WHO-recommended Cq value of 25 was applied across the laboratories, one cohort (United Kingdom) showed clinical sensitivity of about 16% while another (Belgium) showed 90%.

“Using Cq cutoffs, as it turns out, isn’t a reproducible metric. This is because the Cq measurements in these laboratories were uncalibrated. We had no standard of measurement across the globe,” Huggett explains. Historically, viruses have been measured in international units or copy numbers, as seen in HIV, hepatitis B, or hepatitis C. Moreover, a control virus of known quantity is run alongside patient samples to calibrate measurements. These practices were missing from SARS-CoV-2 measurements.

Using a better metric to inform clinical decisions: Copies/mL

To evaluate traditional measurement approaches used in viral quantitation, the three laboratories each generated copy number metrics using an external control. These values, along with control values generated by over 600 other laboratories across the globe, were used to convert patient datasets into copies/mL. When compared using copies/mL, the inter-laboratory variability was reduced. “There's still considerable variation associated with specimen choice, complexity and sampling, but we are much better placed to understand it,” says Huggett. “Using standardized copies/mL removed most of the laboratory technical variation seen with Cq metrics, bringing down the differences to ~10-fold. That type of variation is generally seen between laboratories, which means copy-based units are more suited for exploring routes to standardize SARS-CoV-2 diagnosis.”

Going forward, Huggett recommends that clinical laboratories define how viral copy units correspond with their protocol’s Cq values. A Cq of 25 in one laboratory, for instance, might be 30 in another, based on the calibration used. Moreover, using copy-based units also makes it easier to communicate results to a physician who has been trained to assess viral diseases using this very metric. Huggett adds: “When a physician is informed that a patient has 10,000 copies of a virus versus 10 million copies, that makes it more clinically tangible than cycles 20 verses 30. So, that’s another reason why we advocate this metric.”

The need for global standardization of metrics

The recent pandemic has highlighted the urgent need to standardize metrics that directly impact patient outcomes. “Molecular biology has seen significant developments in being able to produce quantitative results. However, reproducibility is not at the center of laboratory considerations,” notes Huggett. “What’s missing from our scientific community at the moment – and what I hope will change in the near future – is that it should be as important to me that you can reproduce my results as it may be to you. This means considering from the outset, things like calibration/normalization and comprehensive reporting of protocols in publications so that other laboratories can replicate our methods. This will be needed if the genomics revolution is to deliver all that is promised.”

Want to know more on this topic?

Join a live Q&A with Prof. Jim Huggett, Prof. Heinz Zeichhardt and Kathryn Harris as they discuss why the choice of unit is important when quantifying virus in the panel discussion: A lesson from COVID-19 on the dangers of using Cq to quantify nucleic acids

References:

Evans D, Cowen S, Kammel M, O'Sullivan DM, Stewart G, Grunert HP, Moran-Gilad J, Verwilt J, In J, Vandesompele J, Harris K, Hong KH, Storey N, Hingley-Wilson S, Dühring U, Bae YK, Foy C, Braybrook J, Zeichhardt H, Huggett JF. The dangers of using Cq to quantify nucleic acid in biological samples; a lesson from COVID-19. Clin Chem. 2021 Oct 11