In the absence of a “gold standard,” researchers are finding a high frequency of false negatives among SARS-CoV-2 RT-PCR tests
Serology tests designed to detect antibodies to the SARS-CoV-2 coronavirus that causes the COVID-19 illness have been dogged by well-publicized questions about accuracy. However, researchers also are raising concerns about the accuracy of molecular diagnostics which claim to detect the actual presence of the coronavirus itself.
“Diagnostic tests, typically involving a nasopharyngeal swab, can be inaccurate in two ways,” said Steven Woloshin, MD, MS, in a news release announcing a new report that “examines challenges and implications of false-negative COVID-19 tests.” Woloshin is an internist, a professor at Dartmouth Institute, and co-director of the Geisel School of Medicine at Dartmouth.
“A false-positive result mistakenly labels a person infected, with consequences including unnecessary quarantine and contact tracing,” he stated in the news release. “False-negative results are far more consequential, because infected persons who might be asymptomatic may not be isolated and can infect others.”
Woloshin led a team of Dartmouth researchers who analyzed two studies from Wuhan, China, and a literature review by researchers in Europe and South America that indicated diagnostic tests for COVID-19 are frequently generating false negatives. The team published their results in the June 5 New England Journal of Medicine (NEJM).
For example, one research team in Wuhan collected samples from 213 hospitalized COVID-19 patients and found that an approved RT-PCR test produced false negatives in 11% of sputum samples, 27% of nasal samples, and 40% of throat samples. Their research was published on the medRxiv preprint server and has not been peer-reviewed.
The literature review Woloshin’s team studied was also published on medRxiv, titled, “False-Negative Results of Initial Rt-PCR Assays for COVID-19: A Systematic Review.” It indicated that the rate of false negatives could be as high as 29%. The authors of the review looked at five studies that had enrolled a total of 957 patients. “The collected evidence has several limitations, including risk of bias issues, high heterogeneity, and concerns about its applicability,” they wrote. “Nonetheless, our findings reinforce the need for repeated testing in patients with suspicion of SARS-Cov-2 infection.”
Another literature review, published in the Annals of Internal Medicine, titled, “Variation in False-Negative Rate of Reverse Transcriptase Polymerase Chain Reaction–Based SARS-CoV-2 Tests by Time Since Exposure,” estimated the probability of false negatives in RT-PCR tests at varying intervals from the time of exposure and symptom onset. For example, the authors found that the median false-negative rate was 38% if a test was performed on the day of symptom onset, versus 20% three days after onset. Their analysis was based on seven studies, five of which were peer-reviewed, with a total of 1330 test samples.
Doctors also are seeing anecdotal evidence of false negatives. For example, clinicians at UC San Diego Health medical center treated a patient with obvious symptoms of COVID-19, but two tests performed on throat samples were negative. However, a third test, using a sample from a bronchial wash, identified the virus, reported Medscape.
Sensitivity and Specificity of COVID-19 Clinical Laboratory Tests
The key measures of test accuracy are sensitivity, which refers to the ability to detect the presence of the virus, and specificity, the ability to determine that the targeted pathogen is not present. “So, a sensitive test is less likely to provide a false-negative result and a specific test is less likely to provide a false-positive result,” wrote Kirsten Meek, PhD, medical writer and editor, in an article for ARUP Laboratories.
“Analytic” sensitivity and specificity “represent the accuracy of a test under ideal conditions in which specimens have been collected from patients with either high viral loads or a complete absence of exposure,” she wrote. However, “sensitivity and specificity under real-world conditions, in which patients are more variable and specimen collection may not be ideal, can often be lower than reported numbers.”
In a statement defending its ID Now molecular point-of-care test, which came under scrutiny during a study of COVID-19 molecular tests by NYU Langone Health, Northwell Health, and Cleveland Clinic, according to MedTech Dive, Abbott Laboratories blamed improper sample collection and handling for highly-publicized false negatives produced by its rapid test. An FDA issued alert about the test on May 14 noted that Abbott had agreed to conduct post-market studies to identify the cause of the false negatives and suggest remedial actions.
Issues with Emergency Use Authorizations
In their NEJM analysis, Woloshin et al point to issues with the FDA’s process for issuing Emergency Use Authorizations (EUAs). For example, they noted variations in how manufacturers are conducting clinical evaluations to determine test performance. “The FDA prefers the use of ‘natural clinical specimens’ but has permitted the use of ‘contrived specimens’ produced by adding viral RNA or inactivated virus to leftover clinical material,” they wrote.
When evaluating clinical performance, manufacturers ordinarily conduct an index test of patients and compare the results with reference-standard test, according to the Dartmouth researchers. For people showing symptoms, the reference standard should be a clinical diagnosis performed by an independent adjudication panel. However, they wrote, “it is unclear whether the sensitivity of any FDA-authorized commercial test has been assessed in this way.” Additionally, a reference standard for determining sensitivity in asymptomatic people “is an unsolved problem that needs urgent attention to increase confidence in test results for contact-tracing or screening purposes.”
“To truly determine false negatives, you need a gold standard test, which is essentially as close to perfect as we can get,” Stephen Rawlings, MD, PhD, (above), a resident physician of internal medicine and infectious diseases fellow at UC San Diego’s Center for AIDS Research (CFAR), who has been working on SARS-CoV-2 test validation since March. “But there just isn’t one yet for coronavirus,” he told Medscape. (Photo copyright: University of California, San Diego.)
Continued adherence to current measures, such as physical distancing and surface disinfection.
Development of highly sensitive and specific tests or combinations of tests to minimize the risk of false-negative results and ongoing transmission based on a false sense of security.
Improved RT-PCR tests and serological assays.
Development and communication of clear risk-stratified protocols for management of negative COVID-19 test results.
“These protocols must evolve as diagnostic test, transmission, and outcome statistics become more available,” they wrote.
Meanwhile, clinical laboratories remain somewhat on their own at selecting which COVID-19 molecular and serology tests they want to purchase and run in their labs. Complicating such decisions is the fact that many of the nation’s most reputable in vitro diagnostics manufacturers cannot produce enough of their COVID-19 tests to meet demand.
Consequently, when looking to purchase tests for SARS-CoV-2, smaller medical laboratory organizations find themselves evaluating COVID-19 kits developed by little-known or even brand-new companies.
Facing a backlog, the state’s public health laboratory turned to the medical laboratory at Dartmouth Hitchcock Medical Center
Much of the attention surrounding the COVID-19
outbreak—the illness caused by the SARS-CoV-2
coronavirus—has focused on large urban areas such as New York City and Los
Angeles. However, the virus is impacting many rural areas as well. This is true
in New Hampshire, where the diagnostic response required close cooperation
between the state’s public health
laboratory and the clinical
laboratory at its lone academic medical center. Their experience offers
lessons for medical
laboratory leaders nationwide.
“When these things happen and you surge beyond what you
could imagine, it’s the relationships with people that matter more than
anything,” said Christine
L. Bean, PhD, Administrator of New
Hampshire Public Health Laboratory Division of Public Health Services , Concord,
N.H., during a recent Dark
Daily webinar, titled, “What Hospital and Health System Labs Need to
Know About Operational Support and Logistics During the COVID-19 Outbreak.”
As Bean explained, during the earliest stages of the
pandemic the “CDC was doing the testing” and the state lab’s role was limited
to submitting samples from patients deemed as “presumptive positives.” Then, on
Feb. 4, the FDA granted an emergency
use authorization (EUA) allowing use of the CDC-developed real-time
reverse transcriptase PCR (RT-PCR) assay by designated labs.
The New Hampshire Public Health Laboratory (NHPHL) received
its first test kit on Feb. 10, Bean said. But the kits were recalled due to
validation problems with one of the reagents. On Feb. 26, the CDC
issued revised test instructions allowing use of the test without the N3
primer and probe set that had caused the early validation issues. The NHPHL
verified the test under the new guidelines and went live on March 2, she said.
However, with a capacity of 150 to 200 tests per day, the
lab wasn’t equipped to handle a large volume. “Much of what we do is really
population-based,” she said. “Most of the time we’re not doing patient
diagnostic testing.”
Christine L. Bean, PhD (left), Administrator of the New Hampshire Public Health Laboratory, and Joel Lefferts, PhD (right), Assistant Professor of Pathology and Laboratory Medicine, and Assistant Director of the Molecular Pathology, at Dartmouth’s Geisel School of Medicine, spoke with Dark Daily’s Editor-in-Chief Robert Michel during a webinar on what hospital and health system labs need to know about operational support and logistics during the COVID-19 outbreak. The webinar can be freely downloaded by clicking here. (Photo copyright: Dark Daily.)
NHPHL Turns to the Medical Laboratory at DHMC-CGHT for
Help
The DHMC-CGHT lab began having its own discussions about
testing in the first week of February, said Joel
A. Lefferts, PhD, HCLD, DABCC, Assistant Professor of Pathology and
Laboratory Medicine and Assistant Director of Molecular Pathology at
Dartmouth’s Geisel School of
Medicine. They were unsure of how much need there would be, but “throughout
the month of February, we started exploring different testing options,” he said
during the Dark Daily webinar.
The Dartmouth-Hitchcock lab team began with the CDC test. However,
Lefferts noted that the initial FDA guidance was “somewhat restrictive” and required
specific RNA extraction
kits and real-time PCR instruments. “If our lab didn’t have the capability to
perform everything exactly as indicated, we would be running it off-label and
would have to possibly submit our own EUA submission to the FDA,” he explained.
Later, though, the FDA and CDC loosened those restrictions and the lab began testing with the CDC assay on March 18, using a Thermo Fisher ABI 7500Dx instrument, Lefferts said. According to Thermo Fisher’s website, the ABI 7500Dx “is a real-time nucleic acid amplification and five-color fluorescence detection system available for in vitro diagnostic use.”
However, Lefferts continued, “we only had one of these
7500Dx instruments, and it was a relatively manual and labor-intensive
process.” It allowed a maximum of 29 samples per run, he said, and took about
five hours to produce results.
Then, the FDA granted an EUA for Abbott’s
m2000 assay, which runs on the company’s m2000rt real-time PCR instrument.
“We were really excited, because we happened to have two of these systems in
our lab,” he said. “We quickly got on the phone and ordered some of these
kits.”
The DHMC-CGHT lab went live with the new system on March 23.
It can handle up to 94 samples per run, said Lefferts, and with two instruments
running from 6 a.m. to 9 p.m., “there’s a potential to do as many as 10 runs
per day.”
This was the system they used to help New Hampshire’s Public
Health Lab with its backlog. “It was unbelievable to see that our backlog could
be really wiped out,” said Bean.
Challenges for Medical Labs
Gearing up for testing in a public health emergency poses
many challenges, Lefferts advised. “You need to look at what instrumentation
you have in your laboratory, what the experience level of your lab team is, how
much space you have, your expected batch size, and your needed turnaround
time.”
The two labs also had to deal with regulatory uncertainty. “This EUA process is something for which we don’t have much experience,” he said. “Trying to juggle CLIA, CAP, the FDA, and possibly state regulations is a bit challenging. You definitely need to do your research and talk to other clinical laboratories that are doing this testing to get advice.”
Lefferts explained that the most significant challenges to
develop and validate a molecular assay for COVID-19 included:
Availability of validation materials.
Obtaining “positive [viral] samples may be a challenge, depending on where you
are and what you have access to,” said Lefferts. However, he credits the FDA
for being “very proactive” in suggesting alternative sources for “viral isolates or genomic RNA that’s been
extracted from some of these viral isolates.”
Availability of collection kits. “We can
do a lot more testing now,” he said, but one bottleneck is the limited
availability of supplies such as nasopharyngeal swabs
and viral transport media. “We’re looking at alternative collection options,”
he said, such as 3D-printed swabs or even Q-tips [household cotton swabs], though
“hopefully it won’t come to that.” The DHMC-CGHT lab also considered producing its
own transport media.
Turnaround times. “Our lab wants to get
those results out as soon as possible,” Lefferts said. “So, we’re looking at
alternative methods to get that testing out sooner.” For example, “do we just
do the SARS-CoV-2 testing on a patient, or do we need to do other influenza and other viral
pathogens,” while also keeping up with other routine testing during the
pandemic?
Staffing issues. “Fatigue is a big issue
with members of our labs who put in lots of extra hours,” he said. The
DHMC-CGHT lab has developed contingency plans in case lab personnel get sick.
This critical information will be highly useful for
Laboratory Directors and Managers, Laboratory Supervisors and Team Leaders,
Integrated Health System Leaders, Hospital Group Leaders, Physicians and
Physician Group Leaders, Phlebotomy Managers, Courier and Logistics Managers,
and Safety and Compliance Managers.
Along with its assessment of the rate of errors in diagnosis, the IOM has a plan to improve, but will doctors accept the IOM’s advice, or continue business as usual?
Diagnostic errors in the American healthcare system is a problem that is now on the radar screen of policymakers at the Institute of Medicine (IOM). Pathologists and clinical laboratory professionals will welcome this development, because recommendations from the IOM carry weight with Congress.
Thus, should the IOM develop specific actions items intended to reduce medical errors, not only are these suggestions likely to involve more effective use of medical laboratory tests by physicians, but there is a strong probability that Congress might eventually write these recommendations into future healthcare legislation.
The Institute of Medicine is a division of the National Academies of Sciences, Engineering, and Medicine. The IOM recently convened a committee that released a list of recommendations to address the problem of diagnostic errors in medicine. Those recommendations, however, are running up against ingrained mindsets and overconfidence on the part of physicians who are reluctant to include decision-support technology in the diagnostic process. (more…)
