News, Analysis, Trends, Management Innovations for
Clinical Laboratories and Pathology Groups

Hosted by Robert Michel

News, Analysis, Trends, Management Innovations for
Clinical Laboratories and Pathology Groups

Hosted by Robert Michel
Sign In

Multiple Studies Raise Questions About Reliability of Clinical Laboratory COVID-19 Diagnostic Tests

In the absence of a “gold standard,” researchers are finding a high frequency of false negatives among SARS-CoV-2 RT-PCR tests

Serology tests designed to detect antibodies to the SARS-CoV-2 coronavirus that causes the COVID-19 illness have been dogged by well-publicized questions about accuracy. However, researchers also are raising concerns about the accuracy of molecular diagnostics which claim to detect the actual presence of the coronavirus itself.

“Diagnostic tests, typically involving a nasopharyngeal swab, can be inaccurate in two ways,” said Steven Woloshin, MD, MS, in a news release announcing a new report that “examines challenges and implications of false-negative COVID-19 tests.” Woloshin is an internist, a professor at Dartmouth Institute, and co-director of the Geisel School of Medicine at Dartmouth.

“A false-positive result mistakenly labels a person infected, with consequences including unnecessary quarantine and contact tracing,” he stated in the news release. “False-negative results are far more consequential, because infected persons who might be asymptomatic may not be isolated and can infect others.”

Woloshin led a team of Dartmouth researchers who analyzed two studies from Wuhan, China, and a literature review by researchers in Europe and South America that indicated diagnostic tests for COVID-19 are frequently generating false negatives. The team published their results in the June 5 New England Journal of Medicine (NEJM).

For example, one research team in Wuhan collected samples from 213 hospitalized COVID-19 patients and found that an approved RT-PCR test produced false negatives in 11% of sputum samples, 27% of nasal samples, and 40% of throat samples. Their research was published on the medRxiv preprint server and has not been peer-reviewed.

The literature review Woloshin’s team studied was also published on medRxiv, titled, “False-Negative Results of Initial Rt-PCR Assays for COVID-19: A Systematic Review.” It indicated that the rate of false negatives could be as high as 29%. The authors of the review looked at five studies that had enrolled a total of 957 patients. “The collected evidence has several limitations, including risk of bias issues, high heterogeneity, and concerns about its applicability,” they wrote. “Nonetheless, our findings reinforce the need for repeated testing in patients with suspicion of SARS-Cov-2 infection.”

Another literature review, published in the Annals of Internal Medicine, titled, “Variation in False-Negative Rate of Reverse Transcriptase Polymerase Chain Reaction–Based SARS-CoV-2 Tests by Time Since Exposure,” estimated the probability of false negatives in RT-PCR tests at varying intervals from the time of exposure and symptom onset. For example, the authors found that the median false-negative rate was 38% if a test was performed on the day of symptom onset, versus 20% three days after onset. Their analysis was based on seven studies, five of which were peer-reviewed, with a total of 1330 test samples.

Doctors also are seeing anecdotal evidence of false negatives. For example, clinicians at UC San Diego Health medical center treated a patient with obvious symptoms of COVID-19, but two tests performed on throat samples were negative. However, a third test, using a sample from a bronchial wash, identified the virus, reported Medscape.

The lesson for clinicians is that they can’t rely solely on test results but must also consider their own observations of the patient, Joshua Metlay, MD, PhD, of Massachusetts General Hospital told Medscape.

Sensitivity and Specificity of COVID-19 Clinical Laboratory Tests

The key measures of test accuracy are sensitivity, which refers to the ability to detect the presence of the virus, and specificity, the ability to determine that the targeted pathogen is not present. “So, a sensitive test is less likely to provide a false-negative result and a specific test is less likely to provide a false-positive result,” wrote Kirsten Meek, PhD, medical writer and editor, in an article for ARUP Laboratories.

“Analytic” sensitivity and specificity “represent the accuracy of a test under ideal conditions in which specimens have been collected from patients with either high viral loads or a complete absence of exposure,” she wrote. However, “sensitivity and specificity under real-world conditions, in which patients are more variable and specimen collection may not be ideal, can often be lower than reported numbers.”

In a statement defending its ID Now molecular point-of-care test, which came under scrutiny during a study of COVID-19 molecular tests by NYU Langone Health, Northwell Health, and Cleveland Clinic, according to MedTech Dive, Abbott Laboratories blamed improper sample collection and handling for highly-publicized false negatives produced by its rapid test. An FDA issued alert about the test on May 14 noted that Abbott had agreed to conduct post-market studies to identify the cause of the false negatives and suggest remedial actions.

Issues with Emergency Use Authorizations

In their NEJM analysis, Woloshin et al point to issues with the FDA’s process for issuing Emergency Use Authorizations (EUAs). For example, they noted variations in how manufacturers are conducting clinical evaluations to determine test performance. “The FDA prefers the use of ‘natural clinical specimens’ but has permitted the use of ‘contrived specimens’ produced by adding viral RNA or inactivated virus to leftover clinical material,” they wrote.

When evaluating clinical performance, manufacturers ordinarily conduct an index test of patients and compare the results with reference-standard test, according to the Dartmouth researchers. For people showing symptoms, the reference standard should be a clinical diagnosis performed by an independent adjudication panel. However, they wrote, “it is unclear whether the sensitivity of any FDA-authorized commercial test has been assessed in this way.” Additionally, a reference standard for determining sensitivity in asymptomatic people “is an unsolved problem that needs urgent attention to increase confidence in test results for contact-tracing or screening purposes.”

Stephen Rawlings, MD, PhD
“To truly determine false negatives, you need a gold standard test, which is essentially as close to perfect as we can get,” Stephen Rawlings, MD, PhD, (above), a resident physician of internal medicine and infectious diseases fellow at UC San Diego’s Center for AIDS Research (CFAR), who has been working on SARS-CoV-2 test validation since March. “But there just isn’t one yet for coronavirus,” he told Medscape. (Photo copyright: University of California, San Diego.)

In a perspective for Mayo Clinic Proceedings, Colin P. West, MD, PhD; Victor M. Montori, MD, MSc; and Priya Sampathkumar, MD, offered four recommendations for addressing concerns about testing accuracy:

  • Continued adherence to current measures, such as physical distancing and surface disinfection.
  • Development of highly sensitive and specific tests or combinations of tests to minimize the risk of false-negative results and ongoing transmission based on a false sense of security.
  • Improved RT-PCR tests and serological assays.
  • Development and communication of clear risk-stratified protocols for management of negative COVID-19 test results.

“These protocols must evolve as diagnostic test, transmission, and outcome statistics become more available,” they wrote.

Meanwhile, clinical laboratories remain somewhat on their own at selecting which COVID-19 molecular and serology tests they want to purchase and run in their labs. Complicating such decisions is the fact that many of the nation’s most reputable in vitro diagnostics manufacturers cannot produce enough of their COVID-19 tests to meet demand.

Consequently, when looking to purchase tests for SARS-CoV-2, smaller medical laboratory organizations find themselves evaluating COVID-19 kits developed by little-known or even brand-new companies.

—Stephen Beale

Related Information:

New Report Examines Challenges and Implications of False-Negative COVID-19 Tests

Questions about COVID-19 Test Accuracy Raised Across the Testing Spectrum

COVID-19 Test Results: Don’t Discount Clinical Intuition

FDA Provides New Tool to Aid Development and Evaluation of Diagnostic Tests That Detect SARS-CoV-2 Infection

EUA Authorized Serology Test Performance

Emergency Use Authorization (EUA) Information and List of All Current EUAs 

Coronavirus (COVID-19) Update: FDA Provides Promised Transparency for Antibody Tests

Understanding Medical Tests: Sensitivity, Specificity, and Positive Predictive Value

Webinar Part 1: Quality Issues Your Clinical Laboratory Should Know Before You Buy or Select COVID-19 Serology Tests

Webinar Part 2: Achieving High Confidence Levels in the Quality and Accuracy of Your Clinical Lab’s Chosen COVID-19 Serology Tests, featuring James Westgard, PhD

With So Many New COVID-19 Serology Tests Obtaining EUAs from the FDA, How Can Clinical Laboratories Identify Tests That Should Perform Reliably?

As federal and state officials ease many regulatory requirements to speed new COVID-19 serology tests to market with minimum data about performance, labs are left with important questions to answer on their own

Every day, elected officials at all levels of government call for a huge expansion of COVID-19 serology testing. But, as most clinical laboratory managers and pathologists know, it is a complex undertaking for a lab to select any serological test, validate it, then run it daily in support of patient care, and have confidence that the results are accurate and reproducible.

Clinical laboratories across the United States understand the volume of testing will be in the tens of millions—even hundreds of millions—of COVID-19 serology tests. That is an important financial opportunity because it gives clinical labs the opportunity to generate some cash flow to offset the 60% decline in daily routine specimens they have experienced since most states enacted shelter-in-place orders in early March.

But this big opportunity to serve physicians and patients with COVID-19 serology testing also comes with equally big risks. There are three major risks a COVID-19 serology testing program that clinical labs must successfully address, otherwise the consequences can be devastating.

Three Major Serology Testing Risks for Clinical Laboratories

Risk one comes during the time when medical laboratories shop for COVID-19 serology tests. As of this writing, about 20 such tests have an emergency use authorization (EUA) with the Food and Drug Administration (FDA) and more are expected to obtain an EUA. As is true with everything in life, not all of these tests will perform equally. The risk to the lab is that it purchases a COVID-19 assay that later proves to be unreliable, despite the lab’s rigorous validation process.

Risk two derives from the fact that new diagnostic methods are being incorporated into the serology tests that companies are submitting to the FDA for an EUA. Although the data submitted to the FDA may indicate acceptable performance to the federal agency, in actual clinical use many unexpected or unknown factors could be recognized which lower confidence that the new method utilized by this particular assay is producing accurate results. That risk would only be recognized downstream from validation and the lab would find itself dealing with unhappy physicians, patients, and employers (who were using the test to check the health of their employees).

Risk three is supply chain risk. Will the manufacturer of the COVID-19 serology test be capable of supplying all of its clinical lab customers with adequate supplies to meet each lab’s demand for this testing? New manufacturers have an unknown track record in their ability to supply their lab customers. But even the largest in vitro diagnostics (IVD) manufacturers may need to ration kits, reagents, and other consumables to the large number of medical laboratories they serve. This happened with the rapid molecular tests for COVID-19. Community laboratories capable of performing these tests could not obtain adequate supplies to serve their client physicians.

Millions Lost on Faulty COVID-19 Serology Test Kits

If there is a fourth major risk to clinical labs performing COVID-19 serology tests for physicians, patients, and employers (who are screening employees in their workplace) it is the negative publicity that can result if a lab’s choice of a COVID-19 serology test ends up generating inaccurate or unreliable test results.

This is a risk not to be ignored. Dark Daily has already written about the global headlines that resulted after both Spain and the United Kingdom spent tens of millions of dollars on COVID-19 serology kits produced by Chinese companies, only to find out that these tests failed to perform at acceptable levels of accuracy. (See, “Chinese Firm to Replace Clinical Laboratory Test Kits After Spanish Health Authorities Report Tests from China’s Shenzen Bioeasy Were Only 30% Accurate,” April 3, 2020.)

The most recent example is here in the United States. On March 27, Abbott Laboratories announced that the FDA had issued an EUA for its Abbott ID NOW platform and its point-of-care rapid molecular test for COVID-19 that could produce results in less than 15 minutes. This made national news and was hailed regularly during the daily White House COVID-19 Task Force briefings.

But then, last week, the ID NOW COVID-19 test was again in the national headlines. For example, CNN published a story on May 14 with the headline, “Abbott’s Fast COVID-19 Test May Miss Too Many Cases, NYU Study Finds,” in which CNN wrote that authors of a study published on bioRxiv titled, “Performance of Abbott ID NOW Rapid SARS-CoV-2 NAAT,” from NYU Langone Health and Grossman School of Medicine in New York City said “the Abbott test was so inaccurate that it was ‘unacceptable’ for use with their patients.” Concerns centered around the true rate of false negatives. Abbott has robustly defended its test and more studies will be forthcoming.

What is important with the examples of Spain, United Kingdom, and a major IVD manufacturer is that news outlets are ready to pounce on any evidence that COVID-19 tests are returning inaccurate or unreliable results. This is a source of risk which every clinical laboratory wants to avoid.

How Clinical Laboratories Can Minimize Risk When Buying COVID-19 Serology Tests

Recognizing that clinical laboratories have been left to their own devices when selecting which of the 20 or so COVID-19 serology tests with EUAs they should buy, validate, and offer to their clients, The Dark Report and its new COVID-19 STAT Intelligence Briefings will present a free webinar titled “Quality Issues Your Clinical Laboratory Should Know Before You Buy or Select COVID-19 Serology Tests,” on Thursday, May 21 at 1 PM Eastern Daylight Time.

This webinar will be conducted by James O. Westgard, PhD, Founder of Westgard QC, and Sten Westgard, Director of Client Services and Technology for Westgard QC.

Sten Westgard of Westgard QC at the podium at LAB QUALITY CONFAB meeting held by THE DARK REPORT.
During their upcoming webinar, James Westgard, PhD (above), and Sten Westgard of Westgard QC will address how clinical laboratory leaders can evaluate different serology COVID-19 tests by: understanding the testing architecture and intended medical use of COVID-19 testing, taking inventory of lab resources; navigating EUA, LDT, and non-EUA regulatory approval; assessing the expected performance of test methods; understanding the critical performance characteristics for COVID-19 testing; and much more. (Photo copyright: Dark Daily.)

This is an exceptional opportunity to gain an inside perspective of how your lab can address the three major risks identified above when selecting a COVID-19 serology test for use in patient care. You’ll gain essential insights about how to assess the public data on tests with an EUA.

This webinar presentation will also discuss how your lab should view all of its COVID-19 testing as a single program. That’s because your lab may test the same patient with a rapid molecular test, then later do serology tests in the days after the patient may have cleared the infection.

Register now for this critical educational opportunity by clicking here or by entering this URL in your web browser (https://www.darkdaily.com/webinar/quality-issues-your-clinical-laboratory-should-know-before-you-buy-or-select-covid-19-serology-tests/).

—Michael McBride

Related Information:

Quality Issues Your Clinical Laboratory Should Know Before You Buy or Select COVID-19 Serology Tests, featuring James Westgard, PhD

Abbott Launches Molecular Point-of-Care Test to Detect Novel Coronavirus in as Little as Five Minutes

FDA EUA: Abbott ID NOW COVID-19 Test

NYU Study: Performance of Abbott ID NOW Rapid SARS-CoV-2 NAAT

Abbott’s Fast Covid-19 Test May Miss Too Many Cases, NYU Study Finds

COVID-19 Triggers a Cash Flow Crash at Clinical Labs Totaling US $5.2 Billion in Past Seven Weeks; Many Labs Are at Brink of Financial Collapse.

25th Annual Executive War College July 14-15, 2020 Hyatt Regency, New Orleans, LA

14th Annual Lab Quality Confab November 17-18, 2020

;