News, Analysis, Trends, Management Innovations for
Clinical Laboratories and Pathology Groups

Hosted by Robert Michel

News, Analysis, Trends, Management Innovations for
Clinical Laboratories and Pathology Groups

Hosted by Robert Michel
Sign In

Multiple Studies Raise Questions About Reliability of Clinical Laboratory COVID-19 Diagnostic Tests

In the absence of a “gold standard,” researchers are finding a high frequency of false negatives among SARS-CoV-2 RT-PCR tests

Serology tests designed to detect antibodies to the SARS-CoV-2 coronavirus that causes the COVID-19 illness have been dogged by well-publicized questions about accuracy. However, researchers also are raising concerns about the accuracy of molecular diagnostics which claim to detect the actual presence of the coronavirus itself.

“Diagnostic tests, typically involving a nasopharyngeal swab, can be inaccurate in two ways,” said Steven Woloshin, MD, MS, in a news release announcing a new report that “examines challenges and implications of false-negative COVID-19 tests.” Woloshin is an internist, a professor at Dartmouth Institute, and co-director of the Geisel School of Medicine at Dartmouth.

“A false-positive result mistakenly labels a person infected, with consequences including unnecessary quarantine and contact tracing,” he stated in the news release. “False-negative results are far more consequential, because infected persons who might be asymptomatic may not be isolated and can infect others.”

Woloshin led a team of Dartmouth researchers who analyzed two studies from Wuhan, China, and a literature review by researchers in Europe and South America that indicated diagnostic tests for COVID-19 are frequently generating false negatives. The team published their results in the June 5 New England Journal of Medicine (NEJM).

For example, one research team in Wuhan collected samples from 213 hospitalized COVID-19 patients and found that an approved RT-PCR test produced false negatives in 11% of sputum samples, 27% of nasal samples, and 40% of throat samples. Their research was published on the medRxiv preprint server and has not been peer-reviewed.

The literature review Woloshin’s team studied was also published on medRxiv, titled, “False-Negative Results of Initial Rt-PCR Assays for COVID-19: A Systematic Review.” It indicated that the rate of false negatives could be as high as 29%. The authors of the review looked at five studies that had enrolled a total of 957 patients. “The collected evidence has several limitations, including risk of bias issues, high heterogeneity, and concerns about its applicability,” they wrote. “Nonetheless, our findings reinforce the need for repeated testing in patients with suspicion of SARS-Cov-2 infection.”

Another literature review, published in the Annals of Internal Medicine, titled, “Variation in False-Negative Rate of Reverse Transcriptase Polymerase Chain Reaction–Based SARS-CoV-2 Tests by Time Since Exposure,” estimated the probability of false negatives in RT-PCR tests at varying intervals from the time of exposure and symptom onset. For example, the authors found that the median false-negative rate was 38% if a test was performed on the day of symptom onset, versus 20% three days after onset. Their analysis was based on seven studies, five of which were peer-reviewed, with a total of 1330 test samples.

Doctors also are seeing anecdotal evidence of false negatives. For example, clinicians at UC San Diego Health medical center treated a patient with obvious symptoms of COVID-19, but two tests performed on throat samples were negative. However, a third test, using a sample from a bronchial wash, identified the virus, reported Medscape.

The lesson for clinicians is that they can’t rely solely on test results but must also consider their own observations of the patient, Joshua Metlay, MD, PhD, of Massachusetts General Hospital told Medscape.

Sensitivity and Specificity of COVID-19 Clinical Laboratory Tests

The key measures of test accuracy are sensitivity, which refers to the ability to detect the presence of the virus, and specificity, the ability to determine that the targeted pathogen is not present. “So, a sensitive test is less likely to provide a false-negative result and a specific test is less likely to provide a false-positive result,” wrote Kirsten Meek, PhD, medical writer and editor, in an article for ARUP Laboratories.

“Analytic” sensitivity and specificity “represent the accuracy of a test under ideal conditions in which specimens have been collected from patients with either high viral loads or a complete absence of exposure,” she wrote. However, “sensitivity and specificity under real-world conditions, in which patients are more variable and specimen collection may not be ideal, can often be lower than reported numbers.”

In a statement defending its ID Now molecular point-of-care test, which came under scrutiny during a study of COVID-19 molecular tests by NYU Langone Health, Northwell Health, and Cleveland Clinic, according to MedTech Dive, Abbott Laboratories blamed improper sample collection and handling for highly-publicized false negatives produced by its rapid test. An FDA issued alert about the test on May 14 noted that Abbott had agreed to conduct post-market studies to identify the cause of the false negatives and suggest remedial actions.

Issues with Emergency Use Authorizations

In their NEJM analysis, Woloshin et al point to issues with the FDA’s process for issuing Emergency Use Authorizations (EUAs). For example, they noted variations in how manufacturers are conducting clinical evaluations to determine test performance. “The FDA prefers the use of ‘natural clinical specimens’ but has permitted the use of ‘contrived specimens’ produced by adding viral RNA or inactivated virus to leftover clinical material,” they wrote.

When evaluating clinical performance, manufacturers ordinarily conduct an index test of patients and compare the results with reference-standard test, according to the Dartmouth researchers. For people showing symptoms, the reference standard should be a clinical diagnosis performed by an independent adjudication panel. However, they wrote, “it is unclear whether the sensitivity of any FDA-authorized commercial test has been assessed in this way.” Additionally, a reference standard for determining sensitivity in asymptomatic people “is an unsolved problem that needs urgent attention to increase confidence in test results for contact-tracing or screening purposes.”

Stephen Rawlings, MD, PhD
“To truly determine false negatives, you need a gold standard test, which is essentially as close to perfect as we can get,” Stephen Rawlings, MD, PhD, (above), a resident physician of internal medicine and infectious diseases fellow at UC San Diego’s Center for AIDS Research (CFAR), who has been working on SARS-CoV-2 test validation since March. “But there just isn’t one yet for coronavirus,” he told Medscape. (Photo copyright: University of California, San Diego.)

In a perspective for Mayo Clinic Proceedings, Colin P. West, MD, PhD; Victor M. Montori, MD, MSc; and Priya Sampathkumar, MD, offered four recommendations for addressing concerns about testing accuracy:

  • Continued adherence to current measures, such as physical distancing and surface disinfection.
  • Development of highly sensitive and specific tests or combinations of tests to minimize the risk of false-negative results and ongoing transmission based on a false sense of security.
  • Improved RT-PCR tests and serological assays.
  • Development and communication of clear risk-stratified protocols for management of negative COVID-19 test results.

“These protocols must evolve as diagnostic test, transmission, and outcome statistics become more available,” they wrote.

Meanwhile, clinical laboratories remain somewhat on their own at selecting which COVID-19 molecular and serology tests they want to purchase and run in their labs. Complicating such decisions is the fact that many of the nation’s most reputable in vitro diagnostics manufacturers cannot produce enough of their COVID-19 tests to meet demand.

Consequently, when looking to purchase tests for SARS-CoV-2, smaller medical laboratory organizations find themselves evaluating COVID-19 kits developed by little-known or even brand-new companies.

—Stephen Beale

Related Information:

New Report Examines Challenges and Implications of False-Negative COVID-19 Tests

Questions about COVID-19 Test Accuracy Raised Across the Testing Spectrum

COVID-19 Test Results: Don’t Discount Clinical Intuition

FDA Provides New Tool to Aid Development and Evaluation of Diagnostic Tests That Detect SARS-CoV-2 Infection

EUA Authorized Serology Test Performance

Emergency Use Authorization (EUA) Information and List of All Current EUAs 

Coronavirus (COVID-19) Update: FDA Provides Promised Transparency for Antibody Tests

Understanding Medical Tests: Sensitivity, Specificity, and Positive Predictive Value

Webinar Part 1: Quality Issues Your Clinical Laboratory Should Know Before You Buy or Select COVID-19 Serology Tests

Webinar Part 2: Achieving High Confidence Levels in the Quality and Accuracy of Your Clinical Lab’s Chosen COVID-19 Serology Tests, featuring James Westgard, PhD

In New Hampshire, Cooperation Was Key to Handling Clinical Laboratory Testing Challenges Posed by the COVID-19 Outbreak

Facing a backlog, the state’s public health laboratory turned to the medical laboratory at Dartmouth Hitchcock Medical Center

Much of the attention surrounding the COVID-19 outbreak—the illness caused by the SARS-CoV-2 coronavirus—has focused on large urban areas such as New York City and Los Angeles. However, the virus is impacting many rural areas as well. This is true in New Hampshire, where the diagnostic response required close cooperation between the state’s public health laboratory and the clinical laboratory at its lone academic medical center. Their experience offers lessons for medical laboratory leaders nationwide.

“When these things happen and you surge beyond what you could imagine, it’s the relationships with people that matter more than anything,” said Christine L. Bean, PhD, Administrator of New Hampshire Public Health Laboratory Division of Public Health Services , Concord, N.H., during a recent Dark Daily webinar, titled, “What Hospital and Health System Labs Need to Know About Operational Support and Logistics During the COVID-19 Outbreak.”

As Bean explained, during the earliest stages of the pandemic the “CDC was doing the testing” and the state lab’s role was limited to submitting samples from patients deemed as “presumptive positives.” Then, on Feb. 4, the FDA granted an emergency use authorization (EUA) allowing use of the CDC-developed real-time reverse transcriptase PCR (RT-PCR) assay by designated labs.

The New Hampshire Public Health Laboratory (NHPHL) received its first test kit on Feb. 10, Bean said. But the kits were recalled due to validation problems with one of the reagents. On Feb. 26, the CDC issued revised test instructions allowing use of the test without the N3 primer and probe set that had caused the early validation issues. The NHPHL verified the test under the new guidelines and went live on March 2, she said.

However, with a capacity of 150 to 200 tests per day, the lab wasn’t equipped to handle a large volume. “Much of what we do is really population-based,” she said. “Most of the time we’re not doing patient diagnostic testing.”

Christine L. Bean, PhD (left), Administrator of the New Hampshire Public Health Laboratory, and Joel Lefferts, PhD (right), Assistant Professor of Pathology and Laboratory Medicine, and Assistant Director of the Molecular Pathology, at Dartmouth’s Geisel School of Medicine, spoke with Dark Daily’s Editor-in-Chief Robert Michel during a webinar on what hospital and health system labs need to know about operational support and logistics during the COVID-19 outbreak. The webinar can be freely downloaded by clicking here. (Photo copyright: Dark Daily.)

NHPHL Turns to the Medical Laboratory at DHMC-CGHT for Help

By April 1, the public health lab had received 3,500 samples for testing, “which is a lot for us,” said Bean. To help with the backlog, they turned to the Laboratory for Clinical Genomics and Advanced Technology at Dartmouth Hitchcock Medical Center (DHMC-CGHT) in Lebanon, N.H.

The DHMC-CGHT lab began having its own discussions about testing in the first week of February, said Joel A. Lefferts, PhD, HCLD, DABCC, Assistant Professor of Pathology and Laboratory Medicine and Assistant Director of Molecular Pathology at Dartmouth’s Geisel School of Medicine. They were unsure of how much need there would be, but “throughout the month of February, we started exploring different testing options,” he said during the Dark Daily webinar.

The Dartmouth-Hitchcock lab team began with the CDC test. However, Lefferts noted that the initial FDA guidance was “somewhat restrictive” and required specific RNA extraction kits and real-time PCR instruments. “If our lab didn’t have the capability to perform everything exactly as indicated, we would be running it off-label and would have to possibly submit our own EUA submission to the FDA,” he explained.

Later, though, the FDA and CDC loosened those restrictions and the lab began testing with the CDC assay on March 18, using a Thermo Fisher ABI 7500Dx instrument, Lefferts said. According to Thermo Fisher’s website, the ABI 7500Dx “is a real-time nucleic acid amplification and five-color fluorescence detection system available for in vitro diagnostic use.”

However, Lefferts continued, “we only had one of these 7500Dx instruments, and it was a relatively manual and labor-intensive process.” It allowed a maximum of 29 samples per run, he said, and took about five hours to produce results.

Then, the FDA granted an EUA for Abbott’s m2000 assay, which runs on the company’s m2000rt real-time PCR instrument. “We were really excited, because we happened to have two of these systems in our lab,” he said. “We quickly got on the phone and ordered some of these kits.”

The DHMC-CGHT lab went live with the new system on March 23. It can handle up to 94 samples per run, said Lefferts, and with two instruments running from 6 a.m. to 9 p.m., “there’s a potential to do as many as 10 runs per day.”

This was the system they used to help New Hampshire’s Public Health Lab with its backlog. “It was unbelievable to see that our backlog could be really wiped out,” said Bean.

Challenges for Medical Labs

Gearing up for testing in a public health emergency poses many challenges, Lefferts advised. “You need to look at what instrumentation you have in your laboratory, what the experience level of your lab team is, how much space you have, your expected batch size, and your needed turnaround time.”

The two labs also had to deal with regulatory uncertainty. “This EUA process is something for which we don’t have much experience,” he said. “Trying to juggle CLIA, CAP, the FDA, and possibly state regulations is a bit challenging. You definitely need to do your research and talk to other clinical laboratories that are doing this testing to get advice.”

Lefferts explained that the most significant challenges to develop and validate a molecular assay for COVID-19 included:

  • Availability of validation materials. Obtaining “positive [viral] samples may be a challenge, depending on where you are and what you have access to,” said Lefferts. However, he credits the FDA for being “very proactive” in suggesting alternative sources for “viral isolates or genomic RNA that’s been extracted from some of these viral isolates.”
  • Availability of collection kits. “We can do a lot more testing now,” he said, but one bottleneck is the limited availability of supplies such as nasopharyngeal swabs and viral transport media. “We’re looking at alternative collection options,” he said, such as 3D-printed swabs or even Q-tips [household cotton swabs], though “hopefully it won’t come to that.” The DHMC-CGHT lab also considered producing its own transport media.
  • Turnaround times. “Our lab wants to get those results out as soon as possible,” Lefferts said. “So, we’re looking at alternative methods to get that testing out sooner.” For example, “do we just do the SARS-CoV-2 testing on a patient, or do we need to do other influenza and other viral pathogens,” while also keeping up with other routine testing during the pandemic?
  • Staffing issues. “Fatigue is a big issue with members of our labs who put in lots of extra hours,” he said. The DHMC-CGHT lab has developed contingency plans in case lab personnel get sick.

The Bean-Lefferts 60-minute webinar was hosted by Dark Daily’s Editor-in-Chief Robert Michel on April 1. It is freely downloadable by clicking here, or by placing this URL in your web browser:

This critical information will be highly useful for Laboratory Directors and Managers, Laboratory Supervisors and Team Leaders, Integrated Health System Leaders, Hospital Group Leaders, Physicians and Physician Group Leaders, Phlebotomy Managers, Courier and Logistics Managers, and Safety and Compliance Managers.

—Stephen Beale

Related Information:

What Hospital and Health System Labs Need to Know About Operational Support and Logistics During the COVID-19 Outbreak

Diagnostic Errors Get the Attention of the Institute of Medicine, Reinforcing Efforts by Nation’s Clinical Pathology Laboratory Scientists to Improve Patient Safety

Along with its assessment of the rate of errors in diagnosis, the IOM has a plan to improve, but will doctors accept the IOM’s advice, or continue business as usual?

Diagnostic errors in the American healthcare system is a problem that is now on the radar screen of policymakers at the Institute of Medicine (IOM). Pathologists and clinical laboratory professionals will welcome this development, because recommendations from the IOM carry weight with Congress.

Thus, should the IOM develop specific actions items intended to reduce medical errors, not only are these suggestions likely to involve more effective use of medical laboratory tests by physicians, but there is a strong probability that Congress might eventually write these recommendations into future healthcare legislation.

The Institute of Medicine is a division of the National Academies of Sciences, Engineering, and Medicine. The IOM recently convened a committee that released a list of recommendations to address the problem of diagnostic errors in medicine. Those recommendations, however, are running up against ingrained mindsets and overconfidence on the part of physicians who are reluctant to include decision-support technology in the diagnostic process. (more…)