New guidelines come on the heels of recommendations covering post-market modifications to AI products, including those incorporated into systems used by clinical laboratories
Artificial intelligence (AI) is booming in healthcare, and as the technology finds its way into more medical devices and clinical laboratory diagnostic test technologies the US Food and Drug Administration (FDA) has stepped up its efforts to provide regulatory guidance for developers of these products. This guidance will have an impact on the development of new lab test technology that uses AI going forward.
In December, the FDA issued finalized recommendations for submitting information about planned modifications to AI-enabled healthcare products. Then, in January, the federal agency issued draft guidance that covers product management and marketing submission more broadly. It is seeking public comments on the latter document through April 7.
“The FDA has authorized more than 1,000 AI-enabled devices through established premarket pathways,” said Troy Tazbaz, director of the Digital Health Center of Excellence at the FDA’s Center for Devices and Radiological Health, in a press release announcing the draft guidance.
This guidance “would be the first to provide total product life cycle recommendations for AI-enabled devices, tying together all design, development, maintenance and documentation recommendations, if and when finalized,” Healthcare IT News reported.
“Today’s draft guidance brings together relevant information for developers, shares learnings from authorized AI-enabled devices, and provides a first point-of-reference for specific recommendations that apply to these devices, from the earliest stages of development through the device’s entire life cycle,” said Troy Tazbaz (above), director of the Digital Health Center of Excellence at the FDA Center for Devices and Radiological Health, in a press release. The new guidance will likely affect the development of new clinical laboratory diagnostic technologies that use AI. (Photo copyright: LinkedIn.)
Engaging with FDA
One key takeaway from the guidance is that manufacturers “should engage with the FDA early to ensure that the testing to support the marketing submission for an AI-enabled device reflects the agency’s total product lifecycle, risk-based approach,” states an analysis from consulting firm Orrick, Herrington and Sutcliffe LLP.
Another key point is transparency, Orrick noted. For example, manufacturers should be prepared to offer details about the inputs and outputs of their AI models and demonstrate “how AI helps achieve a device’s intended use.”
Manufacturers should also take steps to avoid bias in data collection for these models. For example, they should gather evidence to determine “whether a device benefits all relevant demographic groups similarly to help ensure that such devices are safe and effective for their intended use,” Orrick said.
New Framework for AI in Drug Development
On the same day that FDA announced the device guidelines, the agency also proposed a framework for regulating use of AI models in developing drugs and biologics.
“AI can be used in various ways to produce data or information regarding the safety, effectiveness, or quality of a drug or biological product,” the federal agency stated in a press release. “For example, AI approaches can be used to predict patient outcomes, improve understanding of predictors of disease progression and process, and analyze large datasets.”
The press release noted that this is the first time the agency has proposed guidance on use of AI in drug development.
These include “bias and reliability problems due to variability in the quality, size, and representativeness of training datasets; the black-box nature of AI models in their development and decision-making; the difficulty of ascertaining the accuracy of a model’s output; and the dangers of data drift and a model’s performance changing over time or across environments. Any of these factors, in FDA’s thinking, could negatively impact the reliability and relevancy of the data sponsors provide FDA.”
The FDA also plans to participate in direct testing of AI-enabled healthcare tools. In October, the FDA and the Department of Veterans Affairs (VA) announced that they will launch “a joint health AI lab to evaluate promising emerging technologies,” according to Nextgov/FCW.
Elnahal said the facility will allow federal agencies and private entities “to test applications of AI in a virtual lab environment.” The goal is to ensure that the tools are safe and effective while adhering to “trustworthy AI principles,” he said.
“It’s essentially a place where you get rapid but effective evaluation—from FDA’s standpoint and from VA’s standpoint—on a potential new application of generative AI to, number one, make sure it works,” he told Nextgov/FCW.
He added that the lab will be set up with safeguards to ensure that the technologies can be tested safely.
“As long as they go through the right security protocols, we’d essentially be inviting parties to test their technology with a fenced off set of VA data that doesn’t have any risk of contagion into our actual live systems, but it’s still informative and simulated,” he told Nextgov/FCW.
There has been an explosion in the use of AI, machine learning, deep learning, and natural language processing in clinical laboratory diagnostic technologies. This is equally true of anatomic pathology, where AI-powered image analysis solutions are coming to market. That two federal agencies are motivated to establish guidelines on working relationships for evaluating the development and use of AI in healthcare settings tells you where the industry is headed.
OIG suggests better use of analytics by CMS could prevent gaming of the system by providers; clinical laboratories can help through test utilization management technology
In 2015, CMS implemented the Hospital-Acquired Condition Reduction Program (HACRP) as part of the Patient Protection and Affordable Care Act (ACA). The HACRP program incentivizes hospitals to lower their HAI rates by adjusting reimbursements according to the inpatient quality reporting (hospital IQR) data provided by the healthcare providers. Hospital IQR data is the basis on which CMS validates a hospital’s HAI rate (among other things CMS is tracking) to determine the hospital’s reimbursement rate for that year.
CMS, in 2016, met its regulatory requirement to validate inpatient quality reporting data;
It reviewed data of 400 randomly selected hospitals as well as 49 hospitals targeted for failing to report half their HAIs, or for low scores in the prior year’s validation process;
However, OIG also reported that CMS did not include hospitals that displayed abnormal data patterns in its targeted sample. Targeting those hospitals, according to the OIG, could identify inaccurate reporting.
CMS staff had identified 96 hospitals with aberrant data patterns, but did not target them for validation—even though the agency can select up to 200 targeted hospitals for review, Becker’s Hospital Review pointed out.
Dollars More Important than Deaths
According to the OIG report, Medicare excluded in its investigation dozens of hospitals with suspected HAI reporting. This is odd since the CMS and the Centers for Disease Control (CDC) apparently are aware that some healthcare providers have manipulated data to improve their quality measure scores and thus increase their reimbursement rates.
“Collecting and analyzing quality data is increasingly central to Medicare programs that link payments to quality and value. Therefore, it is important for CMS to ensure that hospitals are not gaming [manipulating data to improve scores] their reporting of quality data,” the OIG report noted.
“There are greater requirements for what a company says about a washing machine’s performance than there is for a hospital on quality of care. And this needs to change,” stated Peter Pronovost, MD, PhD, in the Kaiser Health News article. “We require auditing of financial data, but we don’t require auditing of healthcare quality data, and that implies that dollars are more important than deaths.” Pronovost is Senior Vice President for Patient Safety and Quality at Johns Hopkins University School of Medicine.
Peter Pronovost, MD, PhD (above) testifying on preventable deaths before the Senate Subcommittee on Primary Health and Aging in 2014. He is Senior Vice President for Patient Safety and Quality at Johns Hopkins University School of Medicine in Baltimore. Pronovost told Kaiser Health News that there are no uniform standards for reviewing data that hospitals report to Medicare. (Photo copyright: US Senate Committee on Health, Education, Labor and Pensions.)
Medicare Missed Hospitals with Suspected HAI Data
CMS should have done an in-depth review of many hospitals that submitted “aberrant data patterns” in 2013 and 2014, the OIG stated in its report. According to a Kaiser HealthNews article, such patterns could include:
A rapid change in results;
Improbably low infection rates; and
Assertions that infections nearly always struck before patients arrived at the hospital.
“There’s a certain amount of blind faith that hospitals are going to tell the truth. It’s a bit much to expect that if they had a bad record they are going to fess up to it,” noted Lisa McGiffert, Director of the Safe Patient Project at Consumers Union, in the Kaiser Health News article.
CMS Needs Better Data Analytics
So, what does the OIG advise CMS to do? The agency called for “better use of analytics to ensure the integrity of hospital-reported quality data.” Specifically, OIG suggested CMS:
Identify hospitals with abnormal percentages of patients who had infections on admission;
Apply risk scores to identify hospitals with high propensity to manipulate reporting;
Use experiences to create and improve models that identify hospitals most likely to game their reporting.
CMS’ Administrator Seema Verma reportedly responded, “We will continue to evaluate the use of better analytics as feasible, based on Medicare’s operational capabilities.”
Medical Laboratory Diagnostic Testing Part of Gaming the System
A 2015 CMS/CDC joint statement noted “three ways that hospitals may be deviating from CDC’s definitions for reportable HAIs,” and two involve diagnostic test ordering. According to the OIG report, they include:
Overculturing: Diagnostic tests may be overutilized by providers in absence of clinical symptoms. Hospitals may use positive results to game their data by claiming infections that appeared days later were present on admission and thus not reportable.
Underculturing: Hospitals underculture when they do not order diagnostic tests in the presence of clinical symptoms. By not ordering the test, the hospital does not learn whether the patient truly has an infection and, therefore, the hospital does not have to report it.
Adjudication: Hospital administrative staff may inappropriately overrule those who report infections. HAIs are, therefore, not shared.
Clinical Laboratories Can Help
One in 25 people each day receives an HAI, CDC estimates. The OIG findings should be a reminder to medical laboratories and pathology groups that quality measures and patient outcomes are often transparent to media, patients, and the public.
One way medical laboratories in hospitals and health systems can help is by investing in utilization management technology and protocols that ensure appropriate lab test utilization. Informing doctors on the availability of appropriate diagnostic tests based on patients’ existing conditions, unique physiologies, or medical histories, could help prevent hospitals from inadvertently or deliberately game the system.
Clearly, transparency in healthcare is increasing. That means there will be more news stories revealing federal agencies’ failures to respond to healthcare data in ways that could have protected patients and the public. Clinical laboratories don’t want to be included in negative reporting.
New strategy by employers and payers encourages patients to choose lower-cost providers, or pay the difference over the price cap
Payers are teaming with employers to steer patients to lower cost providers. Their common goal is to reduce the cost of care without compromising the quality of care delivered to their beneficiaries. This trend may involve clinical laboratories and anatomic pathology groups, particularly where a lab is seen as a high-cost provider in its service area.
There is credible evidence that patients are willing to consider lower-cost providers. For example, a pilot project aimed a cutting the cost for knee and hip surgeries saved $5.5 million for the California Public Employees Retirement System (CalPERS), the nation’s largest pension fund and third largest purchaser of healthcare benefits. (more…)
Proteus Biomedical, Inc. prepares to launch a “smart pill” to remotely monitor how medication affects patients.
If some experts are correct, it won’t take long to create ingestible devices that are capable of conducting clinical laboratory tests within the body. These devices would transmit the laboratory test results to physicians over the Internet by using wireless technology.
As soon as 2011, Proteus Medical, Inc., of Redwood City, California, says it expects to introduce an ingestible device for managing heart disease and chronic disease to the clinical market. Proteus named this device the Raisin System and a popular term for this type of technology is “smart pill.” (more…)
Influenza Outbreak Calls Attention to Shortage of Medical Technologists, other lab staff
It took the threat of an influenza pandemic recently to get at least one news reporter to realize the shortage of medical laboratory technicians has reached epidemic proportions.
While the recent outbreak of A/H1N1 influenza turned out to be a dress rehearsal, it inspired Wall Street Journal (WSJ) reporter Laura Landro to focus on the critical role played by medical technologists, clinical laboratory scientists, medical laboratory technicians, and other lab professionals, along with the potential consequences of this clinical laboratory staffing shortage when a killer bug turns out to be “for real.”
