New guidelines come on the heels of recommendations covering post-market modifications to AI products, including those incorporated into systems used by clinical laboratories
Artificial intelligence (AI) is booming in healthcare, and as the technology finds its way into more medical devices and clinical laboratory diagnostic test technologies the US Food and Drug Administration (FDA) has stepped up its efforts to provide regulatory guidance for developers of these products. This guidance will have an impact on the development of new lab test technology that uses AI going forward.
In December, the FDA issued finalized recommendations for submitting information about planned modifications to AI-enabled healthcare products. Then, in January, the federal agency issued draft guidance that covers product management and marketing submission more broadly. It is seeking public comments on the latter document through April 7.
“The FDA has authorized more than 1,000 AI-enabled devices through established premarket pathways,” said Troy Tazbaz, director of the Digital Health Center of Excellence at the FDA’s Center for Devices and Radiological Health, in a press release announcing the draft guidance.
This guidance “would be the first to provide total product life cycle recommendations for AI-enabled devices, tying together all design, development, maintenance and documentation recommendations, if and when finalized,” Healthcare IT News reported.
“Today’s draft guidance brings together relevant information for developers, shares learnings from authorized AI-enabled devices, and provides a first point-of-reference for specific recommendations that apply to these devices, from the earliest stages of development through the device’s entire life cycle,” said Troy Tazbaz (above), director of the Digital Health Center of Excellence at the FDA Center for Devices and Radiological Health, in a press release. The new guidance will likely affect the development of new clinical laboratory diagnostic technologies that use AI. (Photo copyright: LinkedIn.)
Engaging with FDA
One key takeaway from the guidance is that manufacturers “should engage with the FDA early to ensure that the testing to support the marketing submission for an AI-enabled device reflects the agency’s total product lifecycle, risk-based approach,” states an analysis from consulting firm Orrick, Herrington and Sutcliffe LLP.
Another key point is transparency, Orrick noted. For example, manufacturers should be prepared to offer details about the inputs and outputs of their AI models and demonstrate “how AI helps achieve a device’s intended use.”
Manufacturers should also take steps to avoid bias in data collection for these models. For example, they should gather evidence to determine “whether a device benefits all relevant demographic groups similarly to help ensure that such devices are safe and effective for their intended use,” Orrick said.
New Framework for AI in Drug Development
On the same day that FDA announced the device guidelines, the agency also proposed a framework for regulating use of AI models in developing drugs and biologics.
“AI can be used in various ways to produce data or information regarding the safety, effectiveness, or quality of a drug or biological product,” the federal agency stated in a press release. “For example, AI approaches can be used to predict patient outcomes, improve understanding of predictors of disease progression and process, and analyze large datasets.”
The press release noted that this is the first time the agency has proposed guidance on use of AI in drug development.
These include “bias and reliability problems due to variability in the quality, size, and representativeness of training datasets; the black-box nature of AI models in their development and decision-making; the difficulty of ascertaining the accuracy of a model’s output; and the dangers of data drift and a model’s performance changing over time or across environments. Any of these factors, in FDA’s thinking, could negatively impact the reliability and relevancy of the data sponsors provide FDA.”
The FDA also plans to participate in direct testing of AI-enabled healthcare tools. In October, the FDA and the Department of Veterans Affairs (VA) announced that they will launch “a joint health AI lab to evaluate promising emerging technologies,” according to Nextgov/FCW.
Elnahal said the facility will allow federal agencies and private entities “to test applications of AI in a virtual lab environment.” The goal is to ensure that the tools are safe and effective while adhering to “trustworthy AI principles,” he said.
“It’s essentially a place where you get rapid but effective evaluation—from FDA’s standpoint and from VA’s standpoint—on a potential new application of generative AI to, number one, make sure it works,” he told Nextgov/FCW.
He added that the lab will be set up with safeguards to ensure that the technologies can be tested safely.
“As long as they go through the right security protocols, we’d essentially be inviting parties to test their technology with a fenced off set of VA data that doesn’t have any risk of contagion into our actual live systems, but it’s still informative and simulated,” he told Nextgov/FCW.
There has been an explosion in the use of AI, machine learning, deep learning, and natural language processing in clinical laboratory diagnostic technologies. This is equally true of anatomic pathology, where AI-powered image analysis solutions are coming to market. That two federal agencies are motivated to establish guidelines on working relationships for evaluating the development and use of AI in healthcare settings tells you where the industry is headed.
Compilation shows US Veterans Administration spent the most at $16B
Clinical laboratory leaders and pathologists will be interested in which hospital systems are making the largest investments in electronic health record (EHR) technologies. Especially considering laboratory information systems (LIS) must interface with these platforms and require extensive reworking when hospitals change their EHRs. For example, hospitals moving to the Epic Systems EHR often require their laboratories to implement the Epic Beaker LIS as well.
According to information sourced by Becker’s Hospital Review, the top 16 hospital systems each spent $500 million or more on EHRs, adding, however, that the information is “not an exhaustive list.”
Number three on the list is Kaiser Permanente which operates multiple hospitals within its nine healthcare networks across the United States serving 12.5 million members. For that reason, its total investment in EHR technology represents a much larger number of hospitals than the other health systems on the list.
Of the 16 providers on the list, 12 installed EHRs provided by Epic Systems of Verona, Wis. Four of the providers implemented EHRs from Oracle Health (formerly Cerner), North Kansas City, Mo., and Meditech of Westwood, Mass.
“Looking forward, there are many advantages in terms of investing in the future and how we will be aligned with technologies including digital and AI applications,” said pathologist Angelique W. Levi, MD (above), vice chair and director of pathology reference services at Yale School of Medicine, in a news release following a site visit to Geisinger Diagnostic Medicine Institute in Danville, Pa., to see Epic Beaker in operation at Geisinger’s clinical laboratory. “But what we gain immediately—having all the patient information accessible in one place in a linked and integrated fashion—is very important.” (Photo copyright: Yale School of Medicine.)
Provider, EHR, Investment
Becker’s list below shows the total amount invested by the 16 healthcare systems was approximately $38.32 billion. The average EHR implementation cost is $2.39 billion for a large healthcare provider.
Becker’s stated they assembled this list from public sources and that there may be other EHR/hospital contracts with a total cost that also would make the list. It is not common to see a list of what hospitals actually spend to acquire and deploy a new EHR.
Epic added 153 hospitals to its client base in 2023. Epic’s EHR competitors—Oracle and Meditech—both experienced declines in client retention rate, Healthcare IT News reported based on the KLAS data.
“Both current and prospective large organization customers are drawn to Epic because they see the vendor as a consistently high performer that provides strong healthcare IT [information technology], quality relationships, and the opportunity to streamline workflows and improve clinicians’ satisfaction,” Healthcare IT News said of the KLAS report’s findings.
In a blog post, authors of the KLAS report explained that in 2023 Oracle added specialty hospital clients and Meditech “saw several new sales” which included healthcare systems and independent providers.
In the next few years, the industry is “ripe for disruption. Another vendor could come in and turn everything on its head,” the KLAS blog article concluded. “Even those who choose Epic want to have more competitive options to choose from.”
Preparing for an LIS Change
Clinical laboratory leaders who may be transitioning their LIS during a new EHR installation may learn from colleagues who completed such an implementation.
Angelique Levi, MD, vice chair and director of pathology reference services at Yale School of Medicine, who was part of the pathology team, noted that one challenge for labs is addressing “information that’s from many different places when we’re talking about cancer care, prognostic testing, and diagnostics.
“It’s become much more complicated to manage all those data points,” she continued. “Without being on an integrated and aligned system, you’re getting pieces of information from different places, but not the ability to have linked and integrated reports in one spot.”
EHR implementations are among the most labor-intensive, expensive projects undertaken by hospitals. Therefore, it is crucial that clinical laboratory and pathology leaders research and learn why an EHR (and possibly LIS) change is needed, what is expected, and when results will be received.
“The SDPR will consolidate geographically fragmented EMR, PAS, and LIMS systems to create a detailed lifelong patient record and deliver cost savings,” NSW Health said in a news release.
NSW Health is the largest public health system in Australia with more than 220 public hospitals, 16 Local Health Districts, and three Specialty Networks. NSW Health Pathology operates more than 60 pathology laboratories (clinical laboratories in the US) and has 150 patient service centers.
“While this initiative will provide untold benefits to all the patients of NSW, we are excited about its potential for improving the health outcomes of our regional patients,” said Andrew Montague (above), former Chief Executive, Central Coast Local Health District in a press release. “By enabling greater collaboration across all local health districts and specialty health networks, the Single Digital Patient Record will provide clinicians with even better tools to keep the patient at the center of everything we do.” This project is more market evidence of the trend to bring clinical laboratory test results from multiple lab sites into a single data repository. (Photo copyright: Coast Community News.)
Cloud-based Realtime Access to Patient Records
Australia has a population of about 26 million and New South Wales, a state on the east coast, is home to more than eight million people. Though the scale of healthcare in Australia is much smaller than in the US, this is still a major project to pull patient data together from all the NSW hospitals, physicians’ offices, and other healthcare providers such as clinical laboratories and pathology practices.
With the change, NSW clinicians will benefit from a cloud–based system offering up real-time access to patients’ medical records, NSW Health Pathology Chief Executive Tracey McCosker told ITnews.
“Patients and our busy staff will benefit from clinical insights gained from the capture of important new data. Our work in pathology is vital to the diagnostic process and developing a statewide laboratory information management system will ensure we provide the best possible services,” McCosker told ITnews.
The KLAS Research report, “US Hospital Market Share 2022,” states that Epic, located in Verona, Wisconsin, has the largest US electronic health record (EHR) market share, Healthgrades noted. According to KLAS:
NSW Health’s decision to engage Epic came after a process involving 350 clinicians, scientists, and technical experts, Zoran Bolevich, MD, Chief Executive of eHealth NSW and NSW Health’s Chief Information Officer, told ITnews.
NSW Health’s Goal for Statewide Digital Patient Record
It was in December 2020 when NSW Health announced its plan to create the SDPR.
“Our vision is to be able to provide a single, holistic, statewide view of every patient—and for that information to be readily accessible to anyone involved in the patient’s care,” Bolevich said in the news release.
The SDPR, according to NSW Health, will address the following:
Challenges:
Current systems not connected statewide.
Inaccessible patient data.
Duplicative data collection.
Gaps in decision-making.
Goals:
Improve health outcomes.
Create patient centricity.
Leverage insights.
NSW’s government has already invested more than $106 million in the SDPR, Healthcare IT News reported.
Other Large EHR Rollouts
NSW Health is not the only large organization to take on such an ambitious project of creating a large-scale digital patient record. And not always to a successful conclusion.
The US Department of Veterans Affairs (VA)—also intent on EHR modernization—recently announced it is suspending roll-out of the Oracle Cerner EHR at VA centers until June 2023 to address technical issues affecting appointments, referrals, and test results.
Four VA centers in Washington, Oregon, and Ohio already went live with the system in 2022.
“We are delaying all future deployments of the new EHR while we fully assess performance and address every concern. Veterans and clinicians deserve a seamless, modernized health record system, and we will not rest until they get it,” said Deputy Secretary of Veterans Affairs Donald Remy, JD, in a news release.
For its part, Oracle Cerner wrote federal lawmakers noting the importance of continuing the project, which will move the VA away from its former VistA health information system.
“Modernization requires change and some short-term pain for the long-term benefits of a modern technology infrastructure,” noted Oracle Cerner Executive Vice President Ken Glueck in the letter, Becker’s Health IT reported. “A modernization project of this scale and scope necessarily involves time to untangle the decades of customized processes established in support of VistA, which inevitably involves challenges.”
NSW Health’s goal is to build a single repository of health information—including lab test results from multiple clinical laboratory sites. When finished NSW Health expects that sharing patient data will contribute to producing better healthcare outcomes.
However, the VA’s experience—and several other similar attempts at large-scale electronic patient record installations—suggest the work ahead will not be easy. But for NSW Health, it may be worth the effort.
VA Office of Inspector General recommends changes in management processes after doctor is sentenced to long federal prison term
In a compelling report, the US Department of Veterans Affairs (VA) Office of Inspector General (OIG) found that a host of management failures and “deficiencies in the facility’s quality management processes” at an Arkansas VA hospital contributed to “thousands of diagnostic errors” throughout the tenure of the facility’s Chief of Pathology and Laboratory Medical Services Robert Morris Levy, MD.
“Any one of these breakdowns could cause harmful results,” the report states. “Occurring together and over an extended period of time, the consequences were devastating, tragic, and deadly.”
The OIG report’s findings on how hospital and laboratory administrators dealt with Levy over the years of his employment at the Fayetteville VA Medical Center demonstrate why clinical and pathology lab leaders need to be constantly vigilant in how various quality and compliance procedures are administered in their laboratories. When people and processes are not meeting acceptable standards, it is patients who are at risk of being harmed.
In January, the federal court in Arkansas sentenced Levy to “240 months in federal prison, followed by three years of supervised release and ordered [him] to pay $497,745.70 in restitution for one count each of mail fraud and involuntary manslaughter,” according to court documents.
In its coverage of the federal case against Robert Morris Levy, MD (above in a jailhouse photo), former Chief of Pathology and Laboratory Medical Services at the Fayetteville VA Medical Center, The Washington Post wrote, “Levy’s supervisors failed to heed early warnings that he was endangering patients and then were slow to act, according to internal VA documents, court filings, and interviews with 20 congressional officials, veterans and current and former VA employees.” Clinical laboratory managers and hospital pathologists would be well advised to study the VA’s conclusions in its recent report. (Photo copyright: The Washington Post.)
VA Pathologist ReceivedMultiple Suspensions, then Termination
Following his removal in April 2018, the OIG assembled a team of pathologists to review nearly 34,000 cases interpreted by Levy since he began working at the VA hospital. They identified more than 3,000 errors, of which 589 were classified as “major diagnostic discrepancies” potentially having a negative impact on patient care.
Of the 589, 34 were deemed serious enough to require institutional disclosures, defined as a discussion with the patient or the patient’s representative revealing “that an adverse event has occurred during the patient’s care that resulted in or is reasonably expected to result in death or serious injury.”
The OIG report cited at least two deaths likely resulting from misdiagnoses.
Levy’s hospital privileges were initially suspended in March 2016 following a blood alcohol test indicating he was legally intoxicated while at work. He was reinstated about six months later after completing a treatment program and agreeing to submit to random drug testing.
His privileges were suspended again in October 2017 after he showed signs of impairment during a hospital committee meeting. He was terminated in April 2018 after he was arrested for allegedly driving while intoxicated (DWI) during work hours.
Federal Court Indicts Levy on Multiple Counts
Shortly after the OIG team began reviewing Levy’s cases, a separate OIG group launched a criminal investigation. Levy admitted to investigators that he had been an alcoholic for 30 years, the report stated, and that he had “purchased a substance, 2-methyl-2-butanol (2M2B), online that could be ingested, was similar to alcohol but more potent, and was not detectable using routine drug and alcohol testing methods.”
Citing the federal indictment, the OIG report noted that Levy passed 42 drug and alcohol tests following his reinstatement at the hospital in 2016.
In August 2019, federal authorities charged Levy with three counts of involuntary manslaughter along with multiple counts of wire fraud, mail fraud, and making false statements. The wire and mail fraud charges were related to his 2M2B purchases.
Levy pleaded guilty in June 2020 and was sentenced on January 22, 2021. In addition to the 20-year prison term, he was ordered to pay approximately $498,000 in restitution to VA. The OIG report noted that Levy has appealed the sentence.
And in “Arkansas Pathologist Faces Three Manslaughter Charges,” Dark Daily’s sister publication, The Dark Report, noted that “The outcome of [the Levy] case could be a precedent that gives other prosecutors the confidence that they can file criminal charges in cases where evidence shows that a pathologist’s actions contributed to diagnostic errors that directly contributed to the death of one or more patients.”
“This sentence should send a strong message that those who abuse their positions of trust in caring for veterans will be held accountable,” said VA Inspector General Michael J. Missal in a federal Department of Justice (DOJ) press release. “Our thoughts are with all those harmed by Dr. Levy’s actions, and we hope they find some small measure of comfort from what happened here today.” (Photo copyright: Military Times.)
OIG Finds Numerous ‘Deficiencies in Quality Management’
In its report, OIG found deficiencies in quality management going back to Levy’s original appointment as Pathology and Laboratory Medical Services Chief.
He was initially hired in September 2005 as a locum tenens (temporary) provider and appointed as full-time service chief a month later. This was despite a DWI conviction from 1996 and a stay of only eight months with his previous employer.
Neither would have barred the doctor as a potential candidate; however, the OIG report states, “the OIG is concerned that a rigorous process was not in place to better evaluate his clinical competency at the time he was hired.”
And that was just the beginning.
In his role as service chief, Levy was responsible for the Path and Lab quality management program with assistance from a subordinate staff pathologist, “which made the process susceptible to subversion,” the report states.
The VHA requires a second pathologist to review certain findings, such as diagnosis of a new cancer malignancy. But in some cases, “it was determined that Dr. Levy was entering concurrence statements into some patients’ electronic health records (EHR) when a second pathologist had not agreed with the interpretation or diagnosis,” the OIG report states.
In addition, second reads sometimes “were communicated by sticky notes, which provided Dr. Levy the opportunity to alter or ignore the results,” the OIG reported.
Inherent Conflict of Interest, Fear of Reprisals, and OIG Recommendations
The periodic privileging process, which grants ongoing hospital privileges, was based in part on a “10% peer review” conducted by the staff pathologist. “The involvement of a subordinate in the peer review process of a supervisor creates an inherent conflict of interest,” the OIG report stated. And in some cases, appraisals of the doctor’s competence came from non-pathologists.
The OIG report suggested that the Veteran’s Health Administration (VHA) re-examine its guidance on the peer review, which requires cases to be randomly selected. Instead, the report suggests that targeting specific kinds of cases, such as those with higher risk of interpretation error, could be more effective in analyzing a pathologist’s performance.
The OIG report also noted failures in dealing with the doctor’s impairment and fostering a “culture of accountability.” Hospital staff, apparently, reported signs of impairment as early as 2014, including incidents when the doctor smelled of alcohol and displayed hand tremors. But hospital leadership failed to “vigorously address allegations of impairment,” the OIG report states. And in interviews with the OIG, some staffers expressed fear of reprisal if they reported what they saw.
The OIG report offers 10 recommendations to the VA, including practices related to hiring processes, the 10% peer review, and alcohol and drug testing. It makes two additional recommendations to the director of the Ozarks VA health system: one related to the credentialing processes and the other aimed at ensuring staff and patients can report concerns without fear of reprisal.
Clinical laboratory managers and hospital pathologists may want to review these recommendations and consider the value of applying them in their own practices.
With improved genetic sequencing comes larger human genome databases that could lead to new diagnostic and therapeutic biomarkers for clinical laboratories
As the COVID-19 pandemic grabbed headlines, the human genome database at the US Department of Veterans Affairs Million Veterans Program (MVP) quietly grew. Now, this wealth of genomic information—as well as data from other large-scale genomic and genetic collections—is expected to produce new biomarkers for clinical laboratory diagnostics and testing.
In December, cancer genomics company Personalis, Inc. (NASDAQ:PSNL) of Menlo Park, Calif., achieved a milestone and delivered its 100,000th whole human genome sequence to the MVP, according to a news release, which also states that Personalis is the sole sequencing provider to the MVP.
The VA’s MVP program, which started in 2011, has 850,000 enrolled veterans and is expected to eventually involve two million people. The VA’s aim is to explore the role genes, lifestyle, and military experience play in health and human illness, notes the VA’s MVP website.
Health conditions affecting veterans the MVP is researching include:
The VA has contracted with Personalis through September 2021, and has invested $175 million, Clinical OMICS reported. Personalis has earned approximately $14 million from the VA. That’s about 76% of the company’s revenue, according to 2nd quarter data, Clinical OMICS noted.
“The VA MVP is the largest whole genome sequencing project in the United States, and this is a significant milestone for both the program and for Personalis,” said John West (above with wife Judy), Founder and CEO of Personalis, in the news release. “Population-scale sequencing projects of this nature represent a cornerstone in our effort to accelerate the advancement of precision medicine across a wide range of disease areas,” he added. (Photo copyright: MIT Technology Review.)
Database of Veterans’ Genomes Used in Current Research
What has the VA gained from their investment so far? An MVP fact sheet states researchers are tapping MVP data for these and other veteran health-related studies:
Differentiating between prostate cancer tumors that require treatment and others that are slow-growing and not life-threatening.
How genetics drives obesity, diabetes, and heart disease.
How data in DNA translates into actual physiological changes within the body.
Gene variations and patients’ response to Warfarin.
NIH Research Program Studies Effects of Genetics on Health
Another research program, the National Institutes of Health’s All of Us study, recently began returning results to its participants who provided blood, urine, and/or saliva samples. The NIH aims to aid research into health outcomes influenced by genetics, environment, and lifestyle, explained a news release. The program, launched in 2018, has biological samples from more than 270,000 people with a goal of one million participants.
“We’re changing the paradigm for research. Participants are our most important partners in this effort, and we know many of them are eager to get their genetic results and learn about the science they’re making possible,” said Josh Denny, MD, CEO of the NIH’s All of Us research program in the news release. Denny, a physician scientist, was Professor of Biomedical Informatics and Medicine, Director of the Center for Precision Medicine and Vice President for Personalized Medicine at Vanderbilt University Medical Center prior to joining the NIH. (Photo copyright: National Institutes of Health.)
Inclusive Data Could Aid Precision Medicine
The news release notes that more than 80% of biological samples in the All of Us database come from people in communities that have been under-represented in biomedical research.
“We need programs like All of Us to build diverse datasets so that research findings ultimately benefit everyone,” said Brad Ozenberger, PhD, All of Us Genomics Program Director, in the news release.
Precision medicine designed for specific healthcare populations is a goal of the All of Us program.
“[All of Us is] beneficial to all Americans, but actually beneficial to the African American race because a lot of research and a lot of medicines that we are taking advantage of today, [African Americans] were not part of the research,” Chris Crawford, All of US Research Study Navigator, told the Birmingham Times. “As [the All of Us study] goes forward and we get a big diverse group of people, it will help as far as making medicine and treatment that will be more precise for us,” he added.
Large Databases Could Advance Care
Genome sequencing technology continues to improve. It is faster, less complicated, and cheaper to sequence a whole human genome than ever before. And the resulting sequence is more accurate.
Thus, as human genome sequencing databases grow, researchers are deriving useful scientific insights from the data. This is relevant for clinical laboratories because the new insights from studying bigger databases of genomic information will produce new diagnostic and therapeutic biomarkers that can be the basis for new clinical laboratory tests as well as useful diagnostic assays for anatomic pathologists.