In the absence of a “gold standard,” researchers are finding a high frequency of false negatives among SARS-CoV-2 RT-PCR tests
Serology tests designed to detect antibodies to the SARS-CoV-2 coronavirus that causes the COVID-19 illness have been dogged by well-publicized questions about accuracy. However, researchers also are raising concerns about the accuracy of molecular diagnostics which claim to detect the actual presence of the coronavirus itself.
“Diagnostic tests, typically involving a nasopharyngeal swab, can be inaccurate in two ways,” said Steven Woloshin, MD, MS, in a news release announcing a new report that “examines challenges and implications of false-negative COVID-19 tests.” Woloshin is an internist, a professor at Dartmouth Institute, and co-director of the Geisel School of Medicine at Dartmouth.
“A false-positive result mistakenly labels a person infected, with consequences including unnecessary quarantine and contact tracing,” he stated in the news release. “False-negative results are far more consequential, because infected persons who might be asymptomatic may not be isolated and can infect others.”
Woloshin led a team of Dartmouth researchers who analyzed two studies from Wuhan, China, and a literature review by researchers in Europe and South America that indicated diagnostic tests for COVID-19 are frequently generating false negatives. The team published their results in the June 5 New England Journal of Medicine (NEJM).
For example, one research team in Wuhan collected samples from 213 hospitalized COVID-19 patients and found that an approved RT-PCR test produced false negatives in 11% of sputum samples, 27% of nasal samples, and 40% of throat samples. Their research was published on the medRxiv preprint server and has not been peer-reviewed.
The literature review Woloshin’s team studied was also published on medRxiv, titled, “False-Negative Results of Initial Rt-PCR Assays for COVID-19: A Systematic Review.” It indicated that the rate of false negatives could be as high as 29%. The authors of the review looked at five studies that had enrolled a total of 957 patients. “The collected evidence has several limitations, including risk of bias issues, high heterogeneity, and concerns about its applicability,” they wrote. “Nonetheless, our findings reinforce the need for repeated testing in patients with suspicion of SARS-Cov-2 infection.”
Another literature review, published in the Annals of Internal Medicine, titled, “Variation in False-Negative Rate of Reverse Transcriptase Polymerase Chain Reaction–Based SARS-CoV-2 Tests by Time Since Exposure,” estimated the probability of false negatives in RT-PCR tests at varying intervals from the time of exposure and symptom onset. For example, the authors found that the median false-negative rate was 38% if a test was performed on the day of symptom onset, versus 20% three days after onset. Their analysis was based on seven studies, five of which were peer-reviewed, with a total of 1330 test samples.
Doctors also are seeing anecdotal evidence of false negatives. For example, clinicians at UC San Diego Health medical center treated a patient with obvious symptoms of COVID-19, but two tests performed on throat samples were negative. However, a third test, using a sample from a bronchial wash, identified the virus, reported Medscape.
Sensitivity and Specificity of COVID-19 Clinical Laboratory Tests
The key measures of test accuracy are sensitivity, which refers to the ability to detect the presence of the virus, and specificity, the ability to determine that the targeted pathogen is not present. “So, a sensitive test is less likely to provide a false-negative result and a specific test is less likely to provide a false-positive result,” wrote Kirsten Meek, PhD, medical writer and editor, in an article for ARUP Laboratories.
“Analytic” sensitivity and specificity “represent the accuracy of a test under ideal conditions in which specimens have been collected from patients with either high viral loads or a complete absence of exposure,” she wrote. However, “sensitivity and specificity under real-world conditions, in which patients are more variable and specimen collection may not be ideal, can often be lower than reported numbers.”
In a statement defending its ID Now molecular point-of-care test, which came under scrutiny during a study of COVID-19 molecular tests by NYU Langone Health, Northwell Health, and Cleveland Clinic, according to MedTech Dive, Abbott Laboratories blamed improper sample collection and handling for highly-publicized false negatives produced by its rapid test. An FDA issued alert about the test on May 14 noted that Abbott had agreed to conduct post-market studies to identify the cause of the false negatives and suggest remedial actions.
Issues with Emergency Use Authorizations
In their NEJM analysis, Woloshin et al point to issues with the FDA’s process for issuing Emergency Use Authorizations (EUAs). For example, they noted variations in how manufacturers are conducting clinical evaluations to determine test performance. “The FDA prefers the use of ‘natural clinical specimens’ but has permitted the use of ‘contrived specimens’ produced by adding viral RNA or inactivated virus to leftover clinical material,” they wrote.
When evaluating clinical performance, manufacturers ordinarily conduct an index test of patients and compare the results with reference-standard test, according to the Dartmouth researchers. For people showing symptoms, the reference standard should be a clinical diagnosis performed by an independent adjudication panel. However, they wrote, “it is unclear whether the sensitivity of any FDA-authorized commercial test has been assessed in this way.” Additionally, a reference standard for determining sensitivity in asymptomatic people “is an unsolved problem that needs urgent attention to increase confidence in test results for contact-tracing or screening purposes.”
“To truly determine false negatives, you need a gold standard test, which is essentially as close to perfect as we can get,” Stephen Rawlings, MD, PhD, (above), a resident physician of internal medicine and infectious diseases fellow at UC San Diego’s Center for AIDS Research (CFAR), who has been working on SARS-CoV-2 test validation since March. “But there just isn’t one yet for coronavirus,” he told Medscape. (Photo copyright: University of California, San Diego.)
Continued adherence to current measures, such as physical distancing and surface disinfection.
Development of highly sensitive and specific tests or combinations of tests to minimize the risk of false-negative results and ongoing transmission based on a false sense of security.
Improved RT-PCR tests and serological assays.
Development and communication of clear risk-stratified protocols for management of negative COVID-19 test results.
“These protocols must evolve as diagnostic test, transmission, and outcome statistics become more available,” they wrote.
Meanwhile, clinical laboratories remain somewhat on their own at selecting which COVID-19 molecular and serology tests they want to purchase and run in their labs. Complicating such decisions is the fact that many of the nation’s most reputable in vitro diagnostics manufacturers cannot produce enough of their COVID-19 tests to meet demand.
Consequently, when looking to purchase tests for SARS-CoV-2, smaller medical laboratory organizations find themselves evaluating COVID-19 kits developed by little-known or even brand-new companies.
Though experts say an antigen test is not as accurate as PCR tests, its low cost, ease of use, and widespread availability make it a boon for clinical labs performing COVID-19 testing
As former FDA commissioner Scott Gottlieb, MD, explained on Face the Nation, “this kind of technology is a real game changer … it’s a very rapid test that could be used in a doctor’s office. Doctors now have about forty thousand of these Sofia machines already installed in their offices … you do a simple nasal swab and the test itself scans for the antigens that the virus produces.
“The test is about 85% sensitive. So, let’s say a hundred people come into a doctor’s office who have COVID-19, eighty-five of them are going to be able to be tested positive with this test very quickly. It’s a cheap test. It’ll probably be about five dollars a test and you can get a result within five minutes … you’re getting a very fast result and you can start to take action immediately.
“The company itself said that they’re going to be able to produce about two hundred thousand of these tests starting right away. But in several weeks, they’ll be able to produce up to 1.5 million a week. So, this dramatically expands our testing capacity as long as doctors are able to run these tests in their offices.”
In an interview on CBS’ Face the Nation, former FDA Commissioner Scott Gottlieb, MD (above), said, “These antigen-based tests aren’t as reliable, meaning they’re not as sensitive. So, they’re going to miss some patients who have COVID. But in the hands of a doctor who already has a high index of suspicion that the patient may have the disease … they allow you to dramatically expand testing. And they’re very cheap. They’re very easy to perform. And again, most doctors have these machines already in their offices.” (Photo copyright: US Food and Drug Administration.)
Other LDTs That Have Received EUAs
Here’s a look at other laboratory-developed tests from major manufacturers that have received emergency-use authorizations from the FDA:
This test is designed for use with Abbott’s m2000 RealTime system, which is installed in about 200 US medical laboratories, the company says. It can run up to 470 patient samples in 24 hours. As of a May 11 statement, the company said it had shipped more than two million tests in the US.
This test is designed for use with Abbott’s Alinity m system, which the company describes as its “most advanced laboratory molecular instrument,” with the ability to run up to 1,080 tests in 24 hours, according to a press release.
This is a rapid test designed for use with the ID Now system, a compact portable instrument for point-of-care settings such as urgent care clinics. As of May 11, Abbott said it had shipped more than 1.7 million tests in the US, and that it planned to increase manufacturing capacity to two million tests per month.
However, the test has encountered some stumbling blocks. On May 14, the FDA issued an alert stating that the ID Now COVID-19 test could produce inaccurate negative results. This came after researchers at NYU Langone Health, Northwell Health, and Cleveland Clinic reported problems with the test, according to MedTech Dive. Abbott issued a statement suggesting that the problems were due to improper sample collection and handling, however, the FDA said that Abbott had agreed to conduct post-market studies to identify the cause of the false negatives and suggest remedial actions.
This is a qualitative test designed to detect the presence of IgG antibodies following a SARS-CoV-2 infection. The FDA authorized use of the assay on Abbott’s Architect i2000SR system in April, and then followed up with a May 11 EUA for its use on the Alinity i system. In a statement, Abbott said it planned to ship 30 million tests globally starting in May.
In a March statement, the FDA touted this as the first point-of-care COVID-19 test to receive an EUA. The company estimates the detection time as approximately 45 minutes. It is designed for use with Cepheid’s GeneXpert Dx diagnostic software and GeneXpert Infinity systems, which have nearly 5,000 US installations, according to a Cepheid statement.
This test runs on Hologic’s Panther system, which, according to a Hologic press release, can provide results in about three hours and run more than 1,000 tests per day. The company claims that more than 1,000 Panther systems are installed in US labs, and that it expects to produce an average of one million tests per week.
Ortho’s antibody test is designed for use with its VITROS XT 7600, 3600, 5600, and ECi/ECiQ immunodiagnostic systems, which, the company says are installed in more than 1,000 US labs. The Total Reagent Pack is a qualitative test that detects the presence of all antibodies against SARS-CoV-2.
On April 24, Ortho announced it had received another FDA EUA, this one for its Anti-SARS-CoV-2 IgG test, which detects the presence of IgG antibodies. In a statement, the company said it expects to produce “several million” IgG tests per month.
This test is designed for use with Roche’s cobas 6800 and 8800 systems. The 6800 can process up to 384 results in an eight-hour shift, Roche says, compared with 1,056 results for the 8800 model. The company says results are available in about 3.5 hours. In a statement, Roche said it planned to ship 400,000 tests per week.
Roche describes this as a qualitative antibody test that can be used on cobas e series immunoassay analyzers. Testing time is 18 minutes. As of May 19, the test was live at more than 20 US labs, “with plans in the next several weeks to increase to more than 200 commercial and hospital lab sites with the ability to perform millions of tests per week,” the company stated in a press release.
It’s likely the FDA will continue to issue emergency-use authorizations as the agency receives more applications from IVD manufacturers.
Medical laboratories are already using gene sequencing as part of a global effort to identify new variants of the coronavirus and their genetic ancestors
Thanks to advances in genetic sequencing technology that enable medical laboratories to sequence organisms faster, more accurately, and at lower cost than ever before, clinical pathology laboratories worldwide are using that capability to analyze the SARS-CoV-2 coronavirus and identify variants as they emerge in different parts of the world.
The US Centers for Disease Control and Prevention (CDC) now plans to harness the power of gene sequencing through a new consortium called SPHERES (SARS-CoV-2 Sequencing for Public Health Emergency Response, Epidemiology, and Surveillance) to “coordinate SARS-CoV-2 sequencing across the United States,” states a CDC news release. The consortium is led by the CDC’s Advanced Molecular Detection (AMD) program and “aims to generate information about the virus that will strengthen COVID-19 mitigation strategies.”
The consortium is comprised of 11 federal agencies, 20 academic institutions, state public health laboratories in 21 states, nine non-profit research organizations, and 14 lab and IVD companies, including:
Abbott Diagnostics
bioMérieux
Color Genomics
Ginkgo Bioworks
IDbyDNA
Illumina
In-Q-Tel
LabCorp
One Codex
Oxford Nanopore Technologies
Pacific Biosciences
Qiagen
Quest Diagnostics
Verily Life Sciences
‘Fundamentally Changing How Public Health Responds’
Gene sequencing and related technologies have “fundamentally changed how public health responds in terms of surveillance and outbreak response,” said Duncan MacCannell, PhD, Chief Science Officer for the CDC’s Office of Advanced Molecular Detection (OAMD), in an April 30 New York Times (NYT) article, which stated that the CDC SPHERES program “will help trace patterns of transmission, investigate outbreaks, and map how the virus is evolving, which can affect a cure.”
The CDC says that rapid DNA sequencing of SARS-CoV-2 will help monitor significant changes in the virus, support contact tracing efforts, provide information for developers of diagnostics and therapies, and “advance public health research in the areas of transmission dynamics, host response, and evolution of the virus.”
The sequencing laboratories in the consortium have agreed to “release their information into the public domain quickly and in a standard way,” the NYT reported, adding that the project includes standards for what types of information medical laboratories should submit, including, “where and when a sample was taken,” and other critical details.
Even in its early phase, the CDC’s SPHERES project has “made a tangible impact in the number of sequences we’re able to deposit and make publicly available on a daily basis,” said Pavitra Roychoudhury, PhD (above), Acting Instructor and Senior Fellow at the University of Washington, and Research Associate at Fred Hutchinson Cancer Research Center, in an e-mail to the NYT. “What we’re essentially doing is reading these small fragments of viral material and trying to jigsaw puzzle the genome together,” said Roychoudhury in an April 28 New York Times article which covered in detail how experts are tracking the coronavirus since it arrived in the US. (Photo copyright: LinkedIn.)
Sharing Data Between Sequencing Laboratories and Biotech Companies
The CDC announced the SPHERES initiative on April 30, although it launched in early April, the NYT reported.
According to the CDC, SPHERES’ objectives include:
To bring together a network of sequencing laboratories, bioinformatics capacity and subject matter expertise under the umbrella of a massive and coordinated public health sequencing effort.
To identify and prioritize capabilities and resource needs across the network and to align sources of federal, non-governmental, and private sector funding and support with areas of greatest impact and need.
To improve coordination of genomic sequencing between institutions and jurisdictions and to enable more resilience across the network.
To champion concepts of openness, standards-based analysis, and rapid data sharing throughout the United States and worldwide during the COVID-19 pandemic response.
To provide a common forum for US public, private, and academic institutions to share protocols, methods, bioinformatics tools, standards, and best practices.
To establish consistent data and metadata standards, including streamlined repository submission processes, sample prioritization criteria, and a framework for shared, privacy-compliant unique case identifiers.
To align with other national sequencing and bioinformatics networks, and to support global efforts to advance the use of standards and open data in public health.
Implications for Developing a Vaccine
As the virus continues to mutate and evolve, one question is whether a vaccine developed for one variant will work on others. However, several experts told The Washington Post that the SARS-CoV-2 coronavirus is relatively stable compared to viruses that cause seasonal flu (influenza).
“At this point, the mutation rate of the virus would suggest that the vaccine developed for SARS-CoV-2 would be a single vaccine, rather than a new vaccine every year like the flu vaccine,” Peter Thielen, a molecular biologist at the Johns Hopkins University Applied Physics Laboratory, told the Washington Post.
Nor, he said, is one variant likely to cause worse clinical outcomes than others. “So far, we don’t have any evidence linking a specific virus [strain] to any disease severity score. Right now, disease severity is much more likely to be driven by other factors.”
Fast improvements in gene sequencing technology have made it faster, more accurate, and cheaper to sequence. Thus, as the COVID-19 outbreak happened, there were many clinical laboratories around the world with the equipment, the staff, and the expertise to sequence the novel coronavirus and watch it mutate from generation to generation and from region to region around the globe. This capability has never been available in outbreaks prior to the current SARS-CoV-2 outbreak.
Annual growth rates of 11% or more is predicted for molecular diagnostics in coming years
Just as consolidation and acquisitions reshaped the in vitro diagnostics (IVD) manufacturing industry and concentrated market share among just a handful of multi-billion dollar IVD giants, a similar consolidation can be seen in the molecular diagnostics sector. Today it is estimated that just nine global companies control 75% of the molecular diagnostics market.
That market concentration means clinical laboratories and anatomic pathology groups have a just a handful of primary vendors from which to purchase many of the molecular diagnostic assays and genetic tests that are used most frequently in clinical care.
Frost and Sullivan published a detailed study about the molecular diagnostics marketplace. The consulting firm estimated that worldwide sales of molecular diagnostics totaled $4.1 billion in 2010. By contrast, total IVD sales globally were about $48 billion in 2010. That total includes the routine, reference, and esoteric test kits, reagents, and supplies used every day by medical laboratories.
Trend is toward increased use of automation, particularly modular and task-targeted solutions, by medical laboratories across the globe
Worldwide, growing numbers of clinical pathology laboratories and medical laboratories are purchasing total laboratory automation (TLA) systems. This is evidence that the latest generation of laboratory automation products are more robust in use compared to prior years and can be purchased at a price that produces an acceptable return on investment (ROI).
However, the acceptance of total laboratory automation in clinical pathology laboratories is not universal. Only in selected countries have significant numbers of medical laboratories embraced laboratory automation as a way to improve productivity, reduce errors, and boost quality.
