News, Analysis, Trends, Management Innovations for
Clinical Laboratories and Pathology Groups

Hosted by Robert Michel

News, Analysis, Trends, Management Innovations for
Clinical Laboratories and Pathology Groups

Hosted by Robert Michel
Sign In

FDA Authorizes First At-Home COVID-19 Antigen Tests, but Roadblocks Remain for “Fast-and-Frequent” Antigen Testing

Developers of medical laboratory tests had high hopes that cheap saliva-based tests would compete with at-home OTC tests that use nasal swabs, but skepticism among scientists continues

Reverse-transcription polymerase chain reaction (RT-PCR) technology has become the standard for clinical laboratory diagnostic testing used to detect the presence of the SARS-CoV-2 coronavirus. However, to enable more widespread testing, some public health experts have called for deployment of cheap, rapid, saliva-based antigen tests that could be self-administered by consumers in their homes.

Despite the technology’s lower sensitivity compared with RT-PCR testing, the idea of “fast-and-frequent” universal antigen testing has gained support as a possible game-changer against the outbreak, the New York Times reported.

The FDA recently took a step in this direction with its first emergency use authorization for the Ellume COVID-19 at-home antigen test. But other developments suggest that these tests may fall short of the lofty vision initially outlined by the experts.

Ellume’s COVID-19 Home Test
Ellume’s COVID-19 Home Test (above) received emergency use authorization from the FDA on December 15. In a press release, Ellume claimed its rapid-antigen test, “demonstrated a sensitivity of 96% and specificity of 100%, and in asymptomatic individuals, the test demonstrated a sensitivity of 91% and specificity of 96%. This level of accuracy across both symptomatic and asymptomatic individuals is crucial in mitigating the spread of an infectious disease like COVID-19.” (Photo copyright: Ellume.)

The Promise of Rapid Antigen COVID-19 Tests

In a column he wrote for Time in July, Ashish K. Jha, MD, MPH, a practicing General Internist and Dean of the Brown University School of Public Health, described the promise of rapid antigen tests. “Imagine spitting on a special strip of paper every morning and being told two minutes later whether you were positive for COVID-19,” he wrote. “If everyone in the United States did this daily, we would dramatically drop our transmission rates and bring the pandemic under control.”

Another advocate for this approach is Michael Mina, MD, PhD, an assistant professor of epidemiology at the Harvard T.H. Chan School of Public Health and a core member of the School’s Center for Communicable Disease Dynamics (CCDD). In a commentary for Time in November he wrote, “Widespread and frequent rapid antigen testing (public health screening to suppress outbreaks) is the best possible tool we have at our disposal today—and we are not using it.”

However, one major issue with antigen testing is sensitivity. “Antigen tests require higher levels of virus than qPCR [quantitative polymerase chain reaction] to return a positive result,” Jha wrote in Time. However, he contends, “the frequency of testing and the speed of results” counter concerns about accuracy.

Even with lower sensitivity, Jha wrote, the quicker test results from antigen tests “would identify viral loads during the most infectious period, meaning those cases we care most about identifying—at the peak period of infectiousness—are less likely to be missed.”

As the FDA explains, RT-PCR molecular tests “detect the virus’ genetic material,” whereas, according to an article published in Nature, titled, “Fast Coronavirus Tests: What They Can and Can’t Do,” antigen tests can “detect specific proteins … on the surface of the virus, and can identify people who are at the peak of infection, when virus levels in the body are likely to be high.”

At-Home Antigen Tests Receive EUAs

The new antigen test developed by Ellume is “the first over-the-counter (OTC) fully at-home diagnostic test for COVID-19,” the FDA said in a press release. The user self-administers a nasal swab and places it in an analyzer connected to a smartphone app. It can deliver results in 20 minutes. The company states that its test has overall sensitivity of 95% and specificity of 97% based on a clinical study of 198 subjects in a simulated home setting.

Jeffrey Shuren, MD, JD, Director of FDA’s Center for Devices and Radiological Health, said in the FDA press release, “This test, like other antigen tests, is less sensitive and less specific than typical molecular tests run in a lab. However, the fact that it can be used completely at home and return results quickly means that it can play an important role in response to the pandemic.”

Ellume expects to deliver about 20 million tests to the US by the end of June 2021. Multiple outlets reported that the test will cost about $30, AP News reported.

Meanwhile, the FDA also authorized at-home use of Abbott’s BinaxNOW rapid antigen test, which was previously authorized for use in point-of-care settings. This test, which requires a prescription, will sell for $25.

In a series of tweets, Harvard’s Mina applauded both moves, but he wrote that they [antigen tests] still fall short of his vision for fast and frequent testing. He described Abbott’s BinaxNOW as “the type of rapid test I have been calling for,” but said he’d like to see tests priced far less and available without a prescription.

Diminishing Prospects for Saliva-based Antigen Tests?

All rapid antigen tests authorized by the FDA so far require nasopharyngeal and/or nasal swab specimens, and it appears that it may be a long time, if ever, before saliva-based antigen tests are available. The New York Times (NYT) reported in October that two companies working on antigen tests—E25Bio and OraSure (NASDAQ:OSUR)—have dropped plans to enable use of saliva.

E25Bio founder Bobby Brooke Herrera, PhD with E25Bio co-founder and Chief Technology Officer Irene Bosch
“If I was placing a bet—which I am, because I’m leading an antigen-based testing company—I would say it’s going to be very difficult for antigen-based testing to work on saliva samples,” E25Bio founder Bobby Brooke Herrera, PhD (above with E25Bio co-founder and Chief Technology Officer Irene Bosch) told the NYT. (Photo copyright: WCVB-TV.)

One advantage of a saliva-based test is that it would be easier to self-administer. “But as they continued to tinker with their tests, researchers at both E25Bio and OraSure found saliva’s performance to be more lackluster than anticipated, and were forced to pivot,” the New York Times reported. Instead, both companies will seek authorization for use of their tests with nasal swabs.

HHS Contract for Antigen Tests Brings High Rates of False Positives

A recent investigative story in ProPublica, titled, “Rapid Testing Is Less Accurate than the Government Wants to Admit,” raised additional questions about rapid antigen testing. In August, the US Department of Health and Human Services announced it had awarded a $760 million contract for 150 million Abbott BinaxNOW tests to be distributed to schools and nursing homes. But later, according to ProPublica, healthcare workers in Nevada and Vermont reported high rates of false positives.

“With the benefit of hindsight, experts said the Trump administration should have released antigen tests primarily to communities with outbreaks instead of expecting them to work just as well in large groups of asymptomatic people,” ProPublica reported. “Understanding they can produce false results; the government could have ensured that clinics had enough for repeat testing to reduce false negatives and access to more precise PCR tests to weed out false positives.”

A few weeks after the reports from Nevada and Vermont, the FDA issued a letter advising clinical laboratories and healthcare providers about the possibility of false positives, along with steps they could take to improve accuracy.

Though some experts remain hopeful about “fast-and-frequent” testing, others are skeptical and say more research is needed to assess the value of this approach. “We are open to thinking outside the box and coming up with new ways to handle this pandemic,” Esther Babady PhD, D(ABMM) of Memorial Sloan Kettering Cancer Center, told the New York Times. However, she added, “the data for that is what’s missing.”

Nevertheless, were at-home rapid saliva-based antigen tests to become a common choice for healthcare consumers, clinical laboratories that perform RT-PCR testing for COVID-19 could see a marked decrease in orders. Thus, regardless of the current state of antigen testing, its development is worth watching.

—Stephen Beale

Related Information:

How We Can Stop the Spread of COVID-19 by Christmas

Over-the-Counter Home Test for COVID-19 Gets US Green Light

‘A Major Breakthrough’: FDA Authorizes Nation’s First At-Home, Over-the-Counter COVID-19 Test

FDA Authorizes First Rapid, Over-the-Counter Home Coronavirus Test

Rapid Testing Is Less Accurate than the Government Wants to Admit

Daily Coronavirus Testing at Home? Many Experts Are Skeptical

Home Tests Could Help in the Fight Against the Coronavirus. So Where Are They?

Screening to Prevent SARS-CoV-2 Outbreaks: Saliva-Based Antigen Testing Is Better than the PCR Swab

America Needs to Radically Rethink Our COVID-19 Testing Approach

Test Sensitivity Is Secondary to Frequency and Turnaround Time for COVID-19 Surveillance

Abbott COVID-19 Tests at Center of Squabble Between Trump Administration, States

Millions of Rapid COVID-19 Antigen Tests May Help Fill the Testing Gap

Does Precision Oncology Offer Patients Superior Treatment Therapies? Anatomic Pathologists Will Be Interested to Learn Why Oncology Experts Disagree

Number of patients eligible for genome-driven oncology therapy is increasing, but the percentage who reportedly benefit from the therapy remains at less than 5%

Advances in precision medicine in oncology (precision oncology) are fueling the need for clinical laboratory companion diagnostic tests that help physicians choose the best treatment protocols. In fact, this is a fast-growing area of clinical diagnostics for the nation’s anatomic pathologists. However, some experts in the field of genome-based cancer treatments disagree over whether such treatments offer more hype than hope.

At an annual meeting of the American Association for Cancer Research (AACR) in Chicago, David Hyman, MD, Chief of Early Drug Development at Memorial Sloan Kettering Cancer Center in New York, and Vinay Prasad, MD, MPH, Hematologist-Oncologist and Associate Professor of Medicine at Oregon Health and Science University (OHSU), squared off.

Science, a journal of the American Association for the Advancement of Science (AAAS), reported that during a panel discussion, titled, “Is Genome-Informed Cancer Medicine Generating Patient Benefit or Just Hype?,” Prasad argued precision oncology benefits far fewer advanced cancer patients than headlines suggest. “When you look at all the data, it’s a sobering picture,” he told the AACR attendees.

To support his claim, Prasad pointed to a study he co-authored that was published in JAMA Oncology, titled, “Estimation of the Percentage of US Patients with Cancer Who Benefit from Genome-Driven Oncology.”

Prasad and his colleagues evaluated 31 US Food and Drug Administration (FDA) approved drugs, which were “genome-targeted” or “genome-informed” for 38 indications between 2006 and 2018. The researchers sought to answer the question, “How many US patients with cancer are eligible for and benefit annually from genome-targeted therapies approved by the US Food and Drug Administration?”

They found that in 2018 only 8.33% of 609,640 patients with metastatic cancer were eligible for genome-targeted therapy—though this was an increase from 5.09% in 2006.

Even more telling from Prasad’s view, his research team concluded that only 4.9% had benefited from such treatments. Prasad’s study found the percentage of patients estimated to have benefited from genome-informed therapy rose from 1.3% in 2006 to 6.62% in 2018.

“Although the number of patients eligible for genome-driven treatment has increased over time, these drugs have helped a minority of patients with advanced cancer,” the researchers concluded. “To accelerate progress in precision oncology, novel trial designs of genomic therapies should be developed, and broad portfolios of drug development, including immunotherapeutic and cytotoxic approaches, should be pursued.”

The graph above is based on data from a study published in Science titled, “Estimation of the Percentage of US Patients With Cancer Who Benefit from Genome-Driven Oncology,” co-authored by Vinay Prasad, MD, MPH, et al. (Image copyright: Science.)

A Value versus Volume Argument?

Hyman, who leads a team of oncologists that conduct dozens of clinical trials and molecularly selected “basket studies” each year, countered Prasad’s assertions by noting the increase in the number of patients who qualify for precision oncology treatments.

As reported in Science, Hyman said during his AACR presentation that Sloan Kettering matched 15% of the 25,000 patients’ tumors it tested with FDA-approved drugs and 10% with drugs in clinical trials.

“I think this is certainly not hype,” he said during the conference.

Hyman added that another 10% to 15% of patient tumors have a DNA change that matches a potential drug tested in animals. He expects “basket” trials to further increase the patient pool by identifying drugs that can work for multiple tumor types.

The US National Institute of Health (NIH) describes “basket studies” as “a new sort of clinical studies to identify patients with the same kind of mutations and treat them with the same drug, irrespective of their specific cancer type. In basket studies, depending on the mutation types, patients are classified into ‘baskets.’ Targeted therapies that block that mutation are then identified and assigned to baskets where patients are treated accordingly.”

Are Expectations of Precision Medicine Exaggerated?

A profile in MIT Technology Review, titled, “The Skeptic: What Precision Medicine Revolution?,” describes Prasad’s reputation as a “professional scold” noting the 36-year-old professor’s “sharp critiques of contemporary biomedical research, including personalized medicine.” Nevertheless, Prasad is not alone in arguing that precision oncology’s promise is often exaggerated.

Following the Obama Administration’s 2015 announcement of its precision medicine initiative, Michael J. Joyner, MD, Professor of Anesthesiology at the Mayo Clinic, penned a New York Times (NYT) editorial in which he cast doubt on the predictive power of genetic variants to improve disease outcomes.

“Like most ‘moonshot’ medical research initiatives, precision medicine is likely to fall short of expectations,” Joyner wrote. “Medical problems and their underlying biology are not linear engineering exercises and solving them is more than a matter of vision, money, and will.”

Recently, he increased his dissent over current perceptions of precision medicine’s value. In a STAT article, titled, “Precision Medicine’s Rosy Predictions Haven’t Come True. We Need Fewer Promises and More Debate,” Joyner and co-author Nigel Paneth, MD, MPH, Professor of Epidemiology and Biostatistics and Pediatrics at Michigan State University, pushed for more debate over the “gene-centric paradigms” that now “pervade biomedical research.”

“Although some niche applications have been found for precision medicine—and gene therapy is now becoming a reality for a few rare diseases—the effects on public health are miniscule while the costs are astronomical,” they wrote.

Hope for Precision Medicine Remains High

However, optimism over precision oncology among some industry leaders has not waned. Cindy Perettie, CEO of molecular information company Foundation Medicine of Cambridge, Mass., argues genome-directed treatments have reached an “inflection point.”

“Personalized cancer treatment is a possibility for more patients than ever thanks to the advent of targeted therapies,” she told Genetic Engineering and Biotechnology News. “With a growing number of new treatments—including two pan-tumor approvals—the need for broad molecular diagnostic tools to match patients with these therapies has never been greater. We continue to advance our understanding of cancer as a disease of the genome—one in which treatment decisions can be informed by insight into the genomic changes that contribute to each patient’s unique cancer.”

Prasad acknowledges genome-driven therapies are beneficial for some cancers. However, he told MIT Technology Review the data doesn’t support the “rhetoric that we’re reaching exponential growth, or that is taking off, or there’s an inflection point” signaling rapid new advancements.

“Right now, we are investing heavily in immunotherapy and heavily in genomic therapy, but in other categories of drugs, such as cytotoxic drugs, we have stopped investigating in them,” he told Medscape Medical News. “But it’s foolish to do this—we need to have the vision to look beyond the fads we live by in cancer medicine and do things in a broader way,” he added.

“So, I support broader funding because you have to sustain efforts even when things are not in vogue if you want to make progress,” Prasad concluded.

Is precision oncology a fad? Dark Daily has covered the advancements in precision medicine extensively over the past decade, and with the launch of our new Precision Medicine Institute website, we plan to continue reporting on further advancements in personalized medicine.

Time will tell if precision oncology can fulfill its promise. If it does, anatomic pathologists will play an important role in pinpointing patients most likely to benefit from genome-driven treatments.

One thing that the debate between proponents of precision medicine in oncology and their critics makes clear is that more and better clinical studies are needed to document the true effectiveness of target therapies for oncology patients. Such evidence will only reinforce the essential role that anatomic pathologists play in diagnosis, guiding therapeutic decisions, and monitoring the progress of cancer patients.

—Andrea Downing Peck

Related Information:

A Cancer Drug Tailored to your Tumor? Experts Trade Barbs over Precision Oncology

Estimation of the Percentage of US Patients with Cancer who Benefit from Genome-Driven Oncology

2020 Vision: Predictions of What May Shape Precision Medicine

Precision Medicine’s Rosy Predictions Haven’t Come True. We Need Fewer Promises and More Debate

The Skeptic: What Precision Medicine Revolution?

‘Moonshot’ Medicine Will Let Us Down

Basket Studies: An Innovative Approach for Oncology Trials

‘Genome-Driven’ Cancer Drugs Treat Small Minority of Patients

Press Release: The All-New Precision Medicine Institute Website Makes Its Debut

Harvard Medical School Study Finds ‘Staggering’ Amounts of Genetic Diversity in Human Microbiome; Might Be Useful in Diagnostics and Precision Medicine

McKinsey and Company Report Highlights Precision Medicine’s Advancements in Integrating Genetic Testing Results with Electronic Medical Records

Precision Medicine’s Most Successful Innovators to Speak in Nashville, including Vanderbilt Univ. Med. Center, Illumina, Geisinger Health, Northwell Health

Targeted Cancer Therapies Bring New Precision Medicine Tools to Anatomic Pathologists and Clinical Laboratories

Controversy Surrounding Memorial Sloan Kettering Cancer Center and Paige.AI Highlights Risks of Data Sharing and Monetization in Anatomic Pathology

Patient privacy, ethics of monetizing not-for-profit data, and questions surrounding industry conflicts appear after the public announcement of an arrangement to grant exclusive access to academic pathology slides and samples

Clinical laboratories and anatomic pathology groups already serve as gatekeepers for a range of medical data used in patient treatments. Glass slides, paraffin-embedded tissue specimens, pathology reports, and autopsy records hold immense value to researchers. The challenge has been how pathologists (and others) in a not-for-profit academic center could set themselves up to potentially profit from their exclusive access to this archived pathology material.

Now, a recent partnership between Memorial Sloan Kettering Cancer Center (MSK) and Paige.AI (a developer of artificial intelligence for pathology) shows how academic pathology laboratories might accomplish this goal and serve a similar gatekeeper role in research and development using the decades of cases in their archives.

The arrangement, however, is not without controversy.

New York Times, ProPublica Report

Following an investigative report from the New York Times (NYT) and ProPublica, pathologists and board members at MSK are under fire from doctors and scientists there who have concerns surrounding ethics, exclusivity, and profiting from data generated by physicians and but owned by MSK.

“Hospital pathologists have strongly objected to the Paige.AI deal, saying it is unfair that the founders received equity stakes in a company that relies on the pathologists’ expertise and work amassed over 60 years. They also questioned the use of patients’ data—even if it is anonymous—without their knowledge in a profit-driven venture,” the NYT article states.

Prominent members of MSK are facing scrutiny from the media and peers—with some relinquishing stakes in Paige.AI—as part of the backlash of the report. This is an example of the perils and PR concerns lab stakeholders might face concerning the safety of data sharing and profits made by medical laboratories and other diagnostics providers using patient data.

Controversy Surrounds Formation of Paige.AI/MSK Partnership

In February 2018, Paige.AI announced closing the deal on a $25-million round of Series A funding, and in gaining exclusive access to 25-million pathology slides and computational pathology intellectual property held by the Department of Pathology at Memorial Sloan Kettering. Coverage by TechCrunch noted that while MSK received an equity stake as part of the licensing agreement, they were not a cash investor.

TechCrunch lists David Klimstra, MD (left), Chairman of the Department of Pathology, MSK, and Thomas Fuchs, Dr.SC (right), Director of Computational Pathology in the Warren Alpert Center for Digital and Computational Pathology at MSK, as co-founders of Paige.AI. (Photo copyrights: New York Times/Thomas Fuchs Lab.)

Creation of the company involved three hospital insiders and three additional board members with the hospital itself established as part owner, according to STAT.

Unnamed officials told the NYT that board members at MSK only invested in Paige.AI after earlier efforts to generate outside interest and investors were unsuccessful. NYT’s coverage also notes experts in non-profit law and corporate governance have raised questions as to compliance with federal and state laws that govern nonprofits in light of the Paige.AI deal.

Growing Privacy Fallout and Potential Pitfalls for Medical Labs

The original September 2018 NYT coverage noted that Klimstra intends to divest his ownership stake in Paige.AI. Later coverage by NYT in October, notes that Democrat Representative Debbie Dingell of Michigan submitted a letter questioning details about patient privacy related to Paige.AI’s access to MSK’s academic pathology resources.

Privacy continues to be a focus for both media and regulatory scrutiny as patient data continues to fill electronic health record (EHR) systems as well as research and commercial databases. Dark Daily recently covered how University of Melbourne researchers demonstrated how easily malicious parties might reidentify deidentified data. (See “Researchers Easily Reidentify Deidentified Patient Records with 95% Accuracy; Privacy Protection of Patient Test Records a Concern for Clinical Laboratories”, October 10, 2018.)

According to the NYT, MSK also issued a memo to employees announcing new restrictions on interactions with for-profit companies with a moratorium on board members investing in or holding board positions in startups created within MSK. The nonprofit further noted it is considering barring hospital executives from receiving compensation for their work on outside boards.

However, MSK told the NYT this only applies to new deals and will not affect the exclusive deal between Paige.AI and MSK.

“We have determined,” MSK wrote, “that when profits emerge through the monetization of our research, financial payments to MSK-designated board members should be used for the benefit of the institution.”

There are no current official legal filings regarding actions against the partnership. Despite this, the arrangement—and the subsequent fallout after the public announcement of the arrangement—serve as an example of pitfalls medical laboratories and other medical service centers considering similar arrangements might face in terms of public relations and employee scrutiny.

Risk versus Reward of Monetizing Pathology Data

While the Paige.AI situation is only one of multiple concerns now facing healthcare teams and board members at MSK, the events are an example of risks pathologists take when playing a role in a commercial enterprise outside their own operations or departments.

In doing so, the pathologists investing in and shaping the deal with Paige.AI brought criticism from reputable sources and negative exposure in major media outlets for their enterprise, themselves, and MSK as a whole. The lesson from this episode is that pathologists should tread carefully when entertaining offers to access the patient materials and data archived by their respective anatomic pathology and clinical laboratory organizations.

—Jon Stone

Related Information:

Sloan Kettering’s Cozy Deal with Start-Up Ignites a New Uproar

Paige.AI Nabs $25M, Inks IP Deal with Sloan Kettering to Bring Machine Learning to Cancer Pathology

Sloan Kettering Executive Turns Over Windfall Stake in Biotech Start-Up

Cancer Center’s Board Chairman Faults Top Doctor over ‘Crossed Lines’

Memorial Sloan Kettering, You’ve Betrayed My Trust

LVHN Patient Data Not Shared with For-Profit Company in Sloan Kettering Trials

Researchers Easily Reidentify Deidentified Patient Records with 95% Accuracy; Privacy Protection of Patient Test Records a Concern for Clinical Laboratories

IBM’s Watson Not Living Up to Hype, Wall Street Journal and Other Media Report; ‘Dr. Watson’ Has Yet to Show It Can Improve Patient Outcomes or Accurately Diagnose Cancer

Wall Street Journal reports IBM losing Watson-for-Oncology partners and clients, but scientists remain confident artificial intelligence will revolutionize diagnosis and treatment of disease

What happens when a healthcare revolution is overhyped? Results fall short of expectations. That’s the diagnosis from the Wall Street Journal (WSJ) and other media outlets five years after IBM marketed its Watson supercomputer as having the potential to “revolutionize” cancer diagnosis and treatment.

The idea that artificial intelligence (AI) could be used to diagnose cancer and identify appropriate therapies certainly carried with it implications for clinical laboratories and anatomic pathologists, which Dark Daily reported as far back as 2012. It also promised to spark rapid growth in precision medicine. For now, though, that momentum may be stalled.

“Watson can read all of the healthcare texts in the world in seconds,” John E. Kelly III, PhD, IBM Senior Vice President, Cognitive Solutions and IBM Research, told Wired in 2011. “And that’s our first priority, creating a ‘Dr. Watson,’ if you will.”

However, despite the marketing pitch, the WSJ investigation published in August claims IBM has fallen far short of that goal during the past seven years. The article states, “More than a dozen IBM partners and clients have halted or shrunk Watson’s oncology-related projects. Watson cancer applications have had limited impact on patients, according to dozens of interviews with medical centers, companies and doctors who have used it, as well as documents reviewed by the Wall Street Journal.”

Anatomic pathologists—who use tumor biopsies to diagnose cancer—have regularly wondered if IBM’s Watson would actually help physicians do a better job in the diagnosis, treatment, and monitoring of cancer patients. The findings of the Wall Street Journal show that Watson has yet to make much of a positive impact when used in support of cancer care.

The WSJ claims Watson often “didn’t add much value” or “wasn’t accurate.” This lackluster assessment is blamed on Watson’s inability to keep pace with fast-evolving treatment guidelines, as well as its inability to accurately evaluate reoccurring or rare cancers. Despite the more than $15 billion IBM has spent on Watson, the WSJ reports there is no published research showing Watson improving patient outcomes.

Lukas Wartman, MD, Assistant Professor, McDonnell Genome Institute at the Washington University School of Medicine in St. Louis, told the WSJ he rarely uses the Watson system, despite having complimentary access. IBM typically charges $200 to $1,000 per patient, plus consulting fees in some cases, for Watson-for-Oncology, the WSJ reported.

“The discomfort that I have—and that others have had with using it—has been the sense that you never know how much faith you can put in those results,” Wartman said.

Rudimentary Not Revolutionary Intelligence, STAT Notes

IBM’s Watson made headlines in 2011 when it won a head-to-head competition against two champions on the game show “Jeopardy.” Soon after, IBM announced it would make Watson available for medical applications, giving rise to the idea of “Dr. Watson.”

In a 2017 investigation, however, published on STAT, Watson is described as in its “toddler stage,” falling far short of IBM’s depiction of Watson as a “digital prodigy.”

“Perhaps the most stunning overreach is in [IBM’s] claim that Watson-for-Oncology, through artificial intelligence, can sift through reams of data to generate new insights and identify, as an IBM sales rep put it, ‘even new approaches’ to cancer care,” the STAT article notes. “STAT found that the system doesn’t create new knowledge and is artificially intelligent only in the most rudimentary sense of the term.”

STAT reported it had taken six years for data engineers and doctors to train Watson in just seven types of cancers and keep the system updated with the latest knowledge.

“It’s been a struggle to update, I’ll be honest,” Mark Kris, MD, oncologist at Memorial Sloan Kettering Cancer Center in New York and lead Watson trainer, told STAT. “Changing the system of cognitive computing doesn’t turn on a dime like that. You have to put in the literature, you have to put in the cases.” (Photo copyright: Physician Education Resource.)

Watson Recommended Unsafe and Incorrect Treatments, STAT Reported

In July 2018, STAT reported that internal documents from IBM revealed Watson had recommended “unsafe and incorrect” cancer treatments.

David Howard, PhD, Professor, Health Policy and Management, Rollins School of Public Health at Emory University, blames Watson’s failure in part to the dearth of high-quality published research available for the supercomputer to analyze.

“IBM spun a story about how Watson could improve cancer treatment that was superficially plausible—there are thousands of research papers published every year and no doctor can read them all,” Howard told “However, the problem is not that there is too much information, but rather there is too little. Only a handful of published articles are high-quality, randomized trials. In many cases, oncologists have to choose between drugs that have never been directly compared in a randomized trial.”

Howard argues the news media needs to do a better job vetting stories touting healthcare breakthroughs.

“Reporters are often susceptible to PR hype about the potential of new technology—from Watson to ‘wearables’—to improve outcomes,” Howard said. “A lot of stories would turn out differently if they asked a simple question: ‘Where is the evidence?’”

Peter Greulich, a retired IBM manager who has written extensively on IBM’s corporate challenges, told STAT that IBM would need to invest more money and people in the Watson project to make it successful—an unlikely possibility in a time of shrinking revenues at the corporate giant.

“IBM ought to quit trying to cure cancer,” he said. “They turned the marketing engine loose without controlling how to build and construct a product.”

AI Could Still Revolutionize Precision Medicine

Despite the recent negative headlines about Watson, AI continues to offer the promise of one day changing how pathologists and physicians work together to diagnose and treat disease. Isaac Kohane, MD, PhD, Chairman of the Biomedical Informatics Program at Harvard Medical School, told Bloomberg that IBM may have oversold Watson, but he predicts AI one day will “revolutionize medicine.”

“It’s anybody’s guess who is going to be the first to the market leader in this space,” he said. “Artificial intelligence and big data are coming to doctors’ offices and hospitals. But it won’t necessarily look like the ads on TV.”

How AI and precision medicine plays out for clinical laboratories and anatomic pathologists is uncertain. Clearly, though, healthcare is on a path toward increased involvement of computerized decision-making applications in the diagnostic process. Regardless of early setbacks, that trend is unlikely to slow. Laboratory managers and pathology stakeholders would be wise to keep apprised of these developments.

—Andrea Downing Peck

Related Information:

IBM’s Watson Supercomputer Recommended ‘Unsafe and Incorrect’ Cancer Treatments, Internal Documents Show

IBM Pitched its Watson Supercomputer as a Revolution in Cancer Care. It’s Nowhere Close

IBM’s Watson Wins Jeopardy! Next Up: Fixing Health Care

IBM’s Watson Supercomputer Wins Practice Jeopardy Round

Memorial Sloan-Kettering Cancer Center, IBM to Collaborate in Applying Watson Technology to Help Oncologists

IBM Has a Watson Dilemma

MD Anderson Cancer Center’s IBM Watson Project Fails, and So Did the Journalism Related to It

What Went Wrong with IBM’s Watson?

IBM’s Watson Failed Against Cancer but AI Still Has Promise

Will IBM’s ‘Watson on Oncology’ Give Oncologists and Pathologists a Useful Tool for Diagnosing and Treating Various Cancer

Pathologists Take Note: IBM’s Watson to Attack Cancer with Help of WellPoint and Cedars-Sinai

Researchers’ Inability to Reproduce Results from Previous Cancer Studies Could Increase Pressure on Clinical Laboratories and Diagnostic Technology Developers

Pathologists in medical laboratories creating laboratory-developed tests (LDTs) should be aware that some in the scientific community want more transparency about technology and methods

Developers of clinical laboratory tests and medical diagnostic technologies might soon be feeling the pressure to increase their push for transparency and standards that ultimately would make replication easier.

That’s thanks to a review project’s inability to reproduce results from three of five high-profile cancer studies.

The review project is called the Reproducibility Project: Cancer Biology and is a collaboration between network provider Science Exchange of Palo Alto, Calif., and the Center for Open Science in Charlottesville, Va. They attempted to independently replicate selected results from high-profile cancer biology papers in an open fashion. (more…)