Might this be a sign that AI platforms like Watson still cannot diagnose the wide range of patients’ conditions as accurately as a board-certified clinical pathologist?
Computer technology evolves so quickly, products often become obsolete before fulfilling their expected potential. Such, apparently, is the case with Watson, the genius artificial intelligence (AI) brainchild of International Business Machines Corp. (IBM) which was going to revolutionize how healthcare providers diagnose disease. In some areas of healthcare, such as analyzing MRIs and X-rays, AI has been a boon. But from a business perspective, Watson has failed to turn a profit for IBM, so it has to go.
In February, The Wall Street Journal (WSJ) reported that IBM is looking to sell its Watson Health unit because it is not profitable, despite bringing in $1 billion annually in revenue. The sale of Watson Health, the article states, would be aligned with IBM’s goal of streamlining the company and focusing its energies on cloud computing and other AI functions. Because one goal of the Watson project was to give physicians a tool to help them diagnose patients more accurately and faster, the problems that prevented Watson from achieving that goal should be of interest to pathologists and clinical laboratory managers, who daily are on the front lines of helping doctors diagnose the most challenging cases.
In a follow-up article, titled, “Potential IBM Watson Health Sale Puts Focus on Data Challenges,” the WSJ wrote, “… some experts found that it can be difficult to apply AI to treating complex medical conditions. Having access to data that represents patient populations broadly has been a challenge, experts told the Journal, and gaps in knowledge about complex diseases may not be fully captured in clinical databases.”
“I believe that we’re many years away from AI products that really positively impact clinical care for many patients,” Bob Kocher, Partner at Venrock, a venture-capital firm that invests in healthcare IT and related services, told the WSJ.
IBM Watson was promoted as a major resource to help improve medical care and support doctors in making more accurate diagnoses. However, in “IBM’s Retreat from Watson Highlights Broader AI Struggles in Health,” the WSJ reported that “IBM spent several billion dollars on acquisitions to build up Watson [Health] … a unit whose marquee product was supposed to help doctors diagnose and cure cancer … A decade later, reality has fallen short of that promise.”
IBM initially created Watson (above center) to be an AI tool capable of a wide variety of applications, starting with answering questions. In January 2011, Watson made headlines when it defeated previous Jeopardy champions Ken Jennings (left) and Brad Rutter (right) on the popular television game show.After its triumph, IBM announced it would transition Watson for use in medical applications and promoted it as a major resource to help improve medical care and support healthcare professionals in making more precise diagnoses. The company called its new division IBM Watson Health and stated that massive data sets would be the key to accomplish its healthcare mission. That same year, The Dark Report had an IBM executive do a presentation about Watson Health at the Executive War College on Laboratory and Pathology Management. (Photo copyright: CBS News.)
Watson’s Successes and Failures in Healthcare
During the years following Watson’s Jeopardy win, Watson Health made some positive advances in the fields of healthcare data analytics, performance measurements, clinical trial recruitment, and healthcare information technology (HIT).
However, Watson Health also experienced some high-profile failures as well. One such failure involved a collaboration with MD Anderson Cancer Center, established in 2013, to help the health systems’ oncologists develop new tools to benefit cancer patients. MD Anderson ended the relationship in 2018 after spending more than $60 million on the project, citing “multiple examples of unsafe and incorrect treatment recommendations,” made by the Watson supercomputer, Healthcare IT News reported.
Watson Health later readjusted the development and sales of its AI drug discovery tools and altered its marketing strategy amid reports of disappointing sales and skepticism surrounding machine learning for medical applications.
Underestimating the Challenge of AI in Healthcare
Since its inception, Watson Health has achieved substantial growth, mainly through a series of acquisitions. Those targeted acquisitions include:
Merge Healthcare, a healthcare imaging software company that was purchased for $1 billion in 2015,
Phytel, a health management software company that was purchased for an undisclosed amount in 2015,
Explorys, a healthcare analytics company that was purchased for an undisclosed amount in 2015, and
Truven Health Analytics, a provider of cloud-based healthcare data, analytics, and insights that was purchased for $2.6 billion in 2016.
“IBM’s Watson Health business came together as a result of several acquisitions,” said Paddy Padmanabhan, founder and CEO of Damo Consulting, a firm that provides digital transformation strategy and advisory services for healthcare organizations. “The decision to sell the business may also have to do with the performance of those units on top of the core Watson platform’s struggles,” he told Healthcare IT News.
It should be noted that these acquisitions involved companies that already had a product in the market which was generating revenue. So, the proposed sale of Watson Health includes not just the original Watson AI product, but the other businesses that IBM put into its Watson Health business division.
Padmanabhan noted that there are many challenges for AI in healthcare and that “historical data is at best a limited guide to the future when diagnosing and treating complex conditions.” He pointed to the failure with MD Anderson (in the use of Watson Health as a resource or tool for diagnosing cancer) was a setback for both IBM and the use of AI in healthcare. However, Padmanabhan is optimistic regarding the future use of AI in healthcare.
“To use an oft-quoted analogy, AI’s performance in healthcare right now is more akin to that of the hedgehog than the fox. The hedgehog can solve for one problem at a time, especially when the problem follows familiar patterns discerned in narrow datasets,” he told Healthcare IT News. “The success stories in healthcare have been in specific areas such as sepsis and readmissions. Watson’s efforts to apply AI in areas such as cancer care may have underestimated the nuances of the challenge.”
Other experts agree that IBM was overly ambitious and overreached with Watson Health and ended up over-promising and under-delivering.
“IBM’s initial approach misfired due to how the solution AI was trained and developed,” Dan Olds, Principal Analyst with Gabriel Consulting Group, told EnterpriseAI. “It didn’t conform well to how doctors work in the real world and didn’t learn from its experiences with real doctors. It was primarily learning from synthetic cases, not real-life cases.”
Was Watson Already Obsolete?
Another issue with Watson was that IBM’s marketing campaign may have exceeded the product’s design capabilities. When Watson was developed, it was built with AI and information technologies (IT) that were already outdated and behind the newest generation of those technologies, noted Tech Republic.
“There were genuine AI innovation triggers at Watson Health in natural language processing and generation, knowledge extraction and management, and similarity analytics,” Jeff Cribbs, Research Vice President at Gartner Research, told Tech Republic. “The hype got ahead of the engineering, as the hype cycle says it almost always will, and some of those struggles became apparent.”
Can Artificial Intelligence Fulfill its Potential in Healthcare?
The fact that IBM is contemplating the sale of Watson Health is another illustration of how difficult it can be to navigate the healthcare industry in the US. It is probable that someday AI could make healthcare diagnostics more accurate and reduce overall costs, however, data challenges still exist and more research and exploration will be needed for AI to fulfill its potential.
“Today’s AI systems are great in beating you at chess or Jeopardy,” Kumar Srinivas, Chief Technology Officer, Health Plans, at NTT DATA Services told Forbes. “But there are major challenges when addressing practical clinical issues that need a bit of explanation as to ‘why.’ Doctors aren’t going to defer to AI-decisions or respond clinically to a list of potential cancer cases if it’s generated from a black box.”
And perhaps that is the biggest challenge of all. For doctors to entrust their patients’ lives to a supercomputer, it better be as good as the hype. But can AI in healthcare ever accomplish that feat?
“AI can work incredibly well when it’s applied to specific use cases,” gastroenterologist Nirav R. Shah, MD, Chief Medical Officer at Sharecare, told Forbes. “With regards to cancer, we’re talking about a constellation of thousands of diseases, even if the focus is on one type of cancer. What we call ‘breast cancer,’ for example, can be caused by many different underlying genetic mutations and shouldn’t really be lumped together under one heading. AI can work well when there is uniformity and large data sets around a simple correlation or association. By having many data points around a single question, neural networks can ‘learn.’ With cancer, we’re breaking several of these principles.”
So, in deciding to divest itself of Watson Health, IBM may simply be as prescient now as it was when it first embraced the concept of AI in healthcare. The tech giant may foresee that AI will likely never replace the human mind of a trained healthcare diagnostician.
If this proves true—at least for several more years—then board-certified clinical pathologists can continue to justifiably refer to themselves as “the doctor’s doctor” because of their skills in diagnosing difficult-to-diagnose patients, and because of their knowledge of which clinical laboratory tests to order and how to interpret those test results.
First used to track cryptocurrencies such as Bitcoin, blockchain is finding its way into tracking and quality control systems in healthcare, including clinical laboratories and big pharma
Four companies were selected by the US Food and Drug Administration (FDA) to participate in a pilot program that will utilize blockchain technology to create a real-time monitoring network for pharmaceutical products. The companies selected by the FDA include: IBM (NYSE:IBM), Merck (NYSE:MRK), Walmart (NYSE:WMT), and KPMG, an international accounting firm. Each company will bring its own distinct expertise to the venture.
This important project to utilize blockchain technologies in
the pharmaceutical distribution chain is another example of prominent
healthcare organizations looking to benefit from blockchain technology.
Clinical laboratories and health insurers also are collaborating on blockchain projects. A recent intelligence briefing from The Dark Report, the sister publication of Dark Daily, describes collaborations between multiple health insurers and Quest Diagnostics to improve their provider directories using blockchain. (See, “Four Insurers, Quest Developing Blockchain,” July 1, 2019.)
Improving Traceability and Security in Healthcare
Blockchain continues to intrigue federal officials, health network administrators, and health information technology (HIT) developers looking for ways to accurately and efficiently track inventory, improve information access and retrieval, and increase the accuracy of collected and stored patient data.
In the FDA’s February press release announcing the pilot program, Scott Gottlieb, MD, who resigned as the FDA’s Commissioner in April, stated, “We’re invested in exploring new ways to improve traceability, in some cases using the same technologies that can enhance drug supply chain security, like the use of blockchain.”
Congress created this latest program, which is part of the federal US Drug Supply Chain Security Act (DSCSA) enacted in 2013, to identify and track certain prescription medications as they are disseminated nationwide. However, once fully tested, similar blockchain systems could be employed in all aspects of healthcare, including clinical laboratories, where critical supplies, fragile specimens, timing, and quality control are all present.
The FDA hopes the electronic framework being tested during
the pilot will help protect consumers from counterfeit, stolen, contaminated, or
harmful drugs, as well as:
reduce the time needed to track and trace
product inventory;
enable timely retrieval of accurate distribution
information;
increase the accuracy of data shared among the
network members; and
help maintain the integrity of products in the
distribution chain, including ensuring products are stored at the correct
temperature.
In the FDA’s February announcement, Scott Gottlieb, MD (above), the FDA Commissioner at that time, said, “For the drug track-and-trace system, our goals are to fully secure electronic product tracing, which provides a step-by-step account of where a drug product has been located and who has handled it, [and] establish a more robust product verification to ensure that a drug product is legitimate and unaltered.” It’s not hard to imagine how such a tracking system would be equally beneficial in clinical laboratories and hospital pathology departments. (Photo copyright: FDA.)
Companies in the FDA’s Blockchain Pilot
IBM, a leading blockchain provider, will serve as the
technology partner on the project. The tech giant has implemented and provided
blockchain applications to clients for years. Its cloud-based platform provides
customers with end-to-end capabilities that enable them to develop, maintain,
and secure their networks.
“Blockchain could provide an important new approach to further improving trust in the biopharmaceutical supply chain,” said Mark Treshock, Global Blockchain Solutions Leader for Healthcare and Life Sciences at IBM, in a news release. “We believe this is an ideal use for the technology because it can not only provide an audit trail that tracks drugs within the supply chain; it can track who has shared data and with whom, without revealing the data itself. Blockchain has the potential to transform how pharmaceutical data is controlled, managed, shared and acted upon throughout the lifetime history of a drug.”
Merck, known as MSD outside of the US and Canada, is
a global pharmaceutical company that researches and develops medications and
vaccines for both human and animal diseases. Merck delivers health solutions to
customers in more than 140 countries across the globe.
“Our supply chain strategy, planning and logistics are built around the customers and patients we serve,” said Craig Kennedy, Senior Vice President, Global Supply Chain Management at Merck, in the IBM news release. “Reliable and verifiable supply helps improve confidence among all the stakeholders—especially patients—while also strengthening the foundation of our business.”
Kennedy added that transparency is one of Merck’s primary
goals in participating in this blockchain project. “If you evaluate today’s
pharmaceutical supply chain system in the US, it’s really a series of handoffs
that are opaque to each other and owned by an individual party,” he said,
adding, “There is no transparency that provides end-to-end capabilities. This
hampers the ability for tracking and tracing within the supply chain.”
Walmart, the world’s largest company by revenue, will
be distributing drugs through their pharmacies and care clinics for the
project. Walmart has successfully experimented using blockchain technology with
other products. It hopes this new collaboration will benefit their customers,
as well.
“With successful blockchain pilots in pork, mangoes, and leafy greens that provide enhanced traceability, we are looking forward to the same success and transparency in the biopharmaceutical supply chain,” said Karim Bennis, Vice President of Strategic Planning of Health and Wellness at Walmart, in the IBM news release. “We believe we have to go further than offering great products that help our customers live better at everyday low prices. Our customers also need to know they can trust us to help ensure products are safe. This pilot, and US Drug Supply Chain Security Act requirements, will help us do just that.”
KPMG, a multi-national professional services network
based in the Netherlands, will be providing knowledge regarding compliance
issues to the venture.
“Blockchain’s innate ability within a private, permissioned
network to provide an ‘immutable record’ makes it a logical tool to deploy to
help address DSCSA compliance requirements,” said Arun Ghosh, US Blockchain
Leader at KPMG, in the IBM news release. “The ability to leverage existing
cloud infrastructure is making enterprise blockchain increasingly affordable
and adaptable, helping drug manufacturers, distributors, and dispensers meet
their patient safety and supply chain integrity goals.”
The FDA’s blockchain project is scheduled to be completed in
the fourth quarter of 2019, with the end results being published in a DSCSA
report. The participating organizations will evaluate the need for and plan any
future steps at that time.
Blockchain is a new and relatively untested technology
within the healthcare industry. However, projects like those supported by the
FDA may bring this technology to the forefront for healthcare organizations,
including clinical laboratories and pathology groups. Once proven, blockchain
technology could have significant benefits for patient data accuracy and
security.
Wall Street Journal reports IBM losing Watson-for-Oncology partners and clients, but scientists remain confident artificial intelligence will revolutionize diagnosis and treatment of disease
What happens when a healthcare revolution is overhyped? Results fall short of expectations. That’s the diagnosis from the Wall Street Journal (WSJ) and other media outlets five years after IBM marketed its Watson supercomputer as having the potential to “revolutionize” cancer diagnosis and treatment.
“Watson can read all of the healthcare texts in the world in seconds,” John E. Kelly III, PhD, IBM Senior Vice President, Cognitive Solutions and IBM Research, told Wired in 2011. “And that’s our first priority, creating a ‘Dr. Watson,’ if you will.”
However, despite the marketing pitch, the WSJ investigation published in August claims IBM has fallen far short of that goal during the past seven years. The article states, “More than a dozen IBM partners and clients have halted or shrunk Watson’s oncology-related projects. Watson cancer applications have had limited impact on patients, according to dozens of interviews with medical centers, companies and doctors who have used it, as well as documents reviewed by the Wall Street Journal.”
Anatomic pathologists—who use tumor biopsies to diagnose cancer—have regularly wondered if IBM’s Watson would actually help physicians do a better job in the diagnosis, treatment, and monitoring of cancer patients. The findings of the Wall Street Journal show that Watson has yet to make much of a positive impact when used in support of cancer care.
The WSJ claims Watson often “didn’t add much value” or “wasn’t accurate.” This lackluster assessment is blamed on Watson’s inability to keep pace with fast-evolving treatment guidelines, as well as its inability to accurately evaluate reoccurring or rare cancers. Despite the more than $15 billion IBM has spent on Watson, the WSJ reports there is no published research showing Watson improving patient outcomes.
“The discomfort that I have—and that others have had with using it—has been the sense that you never know how much faith you can put in those results,” Wartman said.
Rudimentary Not Revolutionary Intelligence, STAT Notes
IBM’s Watson made headlines in 2011 when it won a head-to-head competition against two champions on the game show “Jeopardy.” Soon after, IBM announced it would make Watson available for medical applications, giving rise to the idea of “Dr. Watson.”
In a 2017 investigation, however, published on STAT, Watson is described as in its “toddler stage,” falling far short of IBM’s depiction of Watson as a “digital prodigy.”
“Perhaps the most stunning overreach is in [IBM’s] claim that Watson-for-Oncology, through artificial intelligence, can sift through reams of data to generate new insights and identify, as an IBM sales rep put it, ‘even new approaches’ to cancer care,” the STAT article notes. “STAT found that the system doesn’t create new knowledge and is artificially intelligent only in the most rudimentary sense of the term.”
STAT reported it had taken six years for data engineers and doctors to train Watson in just seven types of cancers and keep the system updated with the latest knowledge.
“IBM spun a story about how Watson could improve cancer treatment that was superficially plausible—there are thousands of research papers published every year and no doctor can read them all,” Howard told HealthNewsReview.org. “However, the problem is not that there is too much information, but rather there is too little. Only a handful of published articles are high-quality, randomized trials. In many cases, oncologists have to choose between drugs that have never been directly compared in a randomized trial.”
Howard argues the news media needs to do a better job vetting stories touting healthcare breakthroughs.
“Reporters are often susceptible to PR hype about the potential of new technology—from Watson to ‘wearables’—to improve outcomes,” Howard said. “A lot of stories would turn out differently if they asked a simple question: ‘Where is the evidence?’”
Peter Greulich, a retired IBM manager who has written extensively on IBM’s corporate challenges, told STAT that IBM would need to invest more money and people in the Watson project to make it successful—an unlikely possibility in a time of shrinking revenues at the corporate giant.
“IBM ought to quit trying to cure cancer,” he said. “They turned the marketing engine loose without controlling how to build and construct a product.”
AI Could Still Revolutionize Precision Medicine
Despite the recent negative headlines about Watson, AI continues to offer the promise of one day changing how pathologists and physicians work together to diagnose and treat disease. Isaac Kohane, MD, PhD, Chairman of the Biomedical Informatics Program at Harvard Medical School, told Bloomberg that IBM may have oversold Watson, but he predicts AI one day will “revolutionize medicine.”
“It’s anybody’s guess who is going to be the first to the market leader in this space,” he said. “Artificial intelligence and big data are coming to doctors’ offices and hospitals. But it won’t necessarily look like the ads on TV.”
How AI and precision medicine plays out for clinical laboratories and anatomic pathologists is uncertain. Clearly, though, healthcare is on a path toward increased involvement of computerized decision-making applications in the diagnostic process. Regardless of early setbacks, that trend is unlikely to slow. Laboratory managers and pathology stakeholders would be wise to keep apprised of these developments.
Data generated by medical laboratories and diagnostic providers takes an increasing role in treatment and precision medicine and allows greater analysis of data and integration of data into the care process
Most anatomic pathologists recognize that the unstructured data that makes up most pathology reports also represents a barrier to more sophisticated use of the information in those pathology reports. One solution is for pathology groups to adopt synoptic reporting as a way to get a pathology report’s essential data into structured fields.
The healthcare marketplace recognizes the value of structured data. In 2012, venture capitalists funded a new company called Flatiron Health. Flatiron’s goal was to access the medical records of cancer patients specifically to extract the relevant—and generally unstructured—data and put it into a structured database. This structured database could then be used to support both research and clinical care for cancer patients.
How valuable is structured healthcare data? Just this February, Roche paid $1.9 billion to acquire Flatiron. At that point, Flatiron had assembled information about the health records of two million cancer patients.
But Roche (ROG.S), recognizing the value of data, was not done. In July, it entered into an agreement to pay $2.4 billion for the remaining shares of cancer-testing company Foundation Medicine that it did not own. Foundation Medicine sequences tumors and uses that genetic data to assist physicians in diagnosing cancer, making treatment decisions, and identifying cancer patients who qualify for specific clinical trials.
Anatomic pathologists play a central role in the diagnosis, treatment, and monitoring of cancer patients. It behooves the pathology profession to recognize that generating, storing, analyzing, and reporting the data generated from examinations of tumor biopsies is a critical success factor moving forward. Otherwise, other players and stakeholders will move past the pathology profession and stake their own claim to capturing, owning, and using that data to add value in patient care.
How Lack of Standards Impact Transfer of Patient Data
DATAMARK Inc., a business process outsourcing (BPO) company headquartered in El Paso, Texas, reports that analysts from Merrill Lynch, Gartner, and IBM estimate unstructured data comprises roughly 80% of the information in the average electronic medical record. This data could be the key to improving outcomes, tailoring precision medicine treatments, or early diagnosis of chronic diseases.
From narrative descriptions of biopsies to dictated entries surrounding preventative care appointments, these entries hold data that might have value but are difficult to collate, organize, or analyze using software or reporting tools.
To further complicate matters, each service provider in a patient’s chain of care might hold different standards or preferred methods for recording data.
“At this point, [standards] are not to a level that helps with the detailed clinical data that we need for the scientific questions we want to ask,” Nikhil Wagle, MD, Assistant Professor of Medicine, Dana-Farber Cancer Institute, Harvard Medical School, and Associate Member, Broad Institute, told the New York Times.
An oncologist at the Dana Farber Cancer Institute in Boston, Wagle and his colleagues are creating a database of metastatic breast cancer patients capable of linking medical records, treatments, and outcomes with their genetic backgrounds and the genetics of their tumors. Despite best efforts, they’ve only collected 450 records for 375 patients in 2.5 years.
Nikhil Wagle, MD (above), Assistant Professor of Medicine, Dana-Farber Cancer Institute, Harvard Medical School, and Associate Member, Broad Institute, is building databases that link patient outcomes and experiences with their EHRs. But sharing that information has proved problematic, he told the New York Times. “Patients are incredibly engaged and excited,” he said, “[But] right now there isn’t a good solution. Even though the patients are saying, ‘I have consented for you to obtain my medical records,’ there is no good way to get them.” (Photo copyright: Dana-Farber Cancer Institute.)
Additionally, once records are obtained, the information—sometimes spanning hundreds of faxed pages—must still be processed into data compatible with Dana-Farber’s database. And updating and maintaining the database requires a full-time staff of experts that must review the information and accurately enter it as required.
When critical concerns arise—such as a cancer diagnosis—information that could yield valuable clues about treatment options and improve outcomes might be held in any number of data silos in any number of formats.
This doesn’t account for the complexity of organizing such information for researchers who are developing new treatments, applying data to less targeted approaches, or dealing with privacy concerns between care providers.
Moving forward, those who can create and interact with data in a way that requires minimal human touch to make it suitable for analysis, further processing, or archiving, could communicate data more effectively and glean value from the growing trove of data silos created by laboratories around the world.
Big Pharma Making Big Bets on Structured Data
These are all the reasons why the recent moves by Roche show the importance and perceived value of structured medical records data as it takes an increasingly important role in precision medicine treatments and diagnosis.
With its acquisition of both Flatiron Health and Foundation Medicine, Roche has secured the ability to generate data, convert said data into a structured format to drive decisions, improve core data-related services, and promote the value of their offerings. This positions Roche to maximize the value of its data for internal use and marketing to researchers and other interested parties.
For clinical laboratories, pathology groups, and other diagnostics providers generating untold amounts of data daily, this highlights a critical opportunity to stay ahead of future trends and position themselves as valuable sources of information as healthcare data continues to play an essential role in modern healthcare.
If this medical imaging collaborative develops a way to use the unstructured data in radiology images and anatomic pathology reports, it could create a new revenue stream for pathologists
Unstructured data has been regularly recognized as one Achilles heel for the anatomic pathology profession. It means invaluable information about the cancers and other diseases diagnosed by surgical pathologists are “locked up,” making it difficult for this information to be accessed in efforts to advance population health management (PHM) or conduct clinical studies.
Similarly, medical imaging has an essential role in the diagnosis of cancer and other diseases. And, like most anatomic pathology reports, medical imaging also is considered to be “unstructured” by data experts because it is not easily accessible by computers, reported Fortune magazine.
Unstructured Data in Anatomic Pathology and Radiology
Now one of the world’s largest information technology companies wants to tackle the challenge of unstructured data in radiology images. IBM (NYSE: IBM) Watson Health launched a global initiative involving 16 health systems, radiology providers, and imaging technology companies.
The Watson Health medical imaging collaborative is working to apply cognitive computing of radiology images to clinical practice. IBM aims to transform how physicians use radiology images to diagnose and monitor patients. (more…)
