New artificial intelligence model agrees with interpretations of human medical technologists and microbiologists with extraordinary accuracy
Microbiology laboratories will be interested in news from Brescia University in Italy, where researchers reportedly have developed a deep learning model that can visually identify and analyze bacterial species in culture plates with a high level of agreement with interpretations made by medical technologists.
They initially trained and tested the system to digitally identify pathogens associated with urinary tract infections (UTIs). UTIs are the source for a large volume of clinical laboratory microbiological testing.
The system, known as DeepColony, uses hierarchical artificial intelligence technology. The researchers say hierarchical AI is better suited to complex decision-making than other approaches, such as generative AI.
In their Nature paper, the researchers explained that microbiologists use conventional methods to visually examine culture plates that contain bacterial colonies. The scientists hypothesize which species of bacteria are present, after which they test their hypothesis “by regrowing samples from each colony separately and then employing mass spectroscopy techniques,” to confirm their hypotheses.
However, DeepColony—which was designed for use with clinical laboratory automation systems—looks at high-resolution digital scans of cultured plates and attempts to identify the bacterial strains and analyze them in much the same way a microbiologist would. For example, it can identify species based on their appearance and determine which colonies are suitable for analysis, the researchers explained.
“Working on a large stream of clinical data, and a complete set of 32 pathogens, the proposed system is capable of effectively assisting plate interpretation with a surprising degree of accuracy in the widespread and demanding framework of urinary tract infections,” the study authors wrote. “Moreover, thanks to the rich species-related generated information, DeepColony can be used for developing trustworthy clinical decision support services in laboratory automation ecosystems from local to global scale.”
“Compared to the most common solutions based on single convolutional neural networks (CNN), multi-network architectures are attractive in our case because of their ability to fit into contexts where decision-making processes are stratified into a complex structure,” wrote the study’s lead author Alberto Signoroni, PhD (above), Associate Professor of Computer Science, University of Brescia, and his researcher team in their Nature paper. “The system must be designed to generate useful and easily interpretable information and to support expert decisions according to safety-by-design and human-in-the-loop policies, aiming at achieving cost-effectiveness and skill-empowerment respectively.” Microbiologists and clinical laboratory managers will want to follow the further development of this technology. (Photo copyright: University of Brescia.)
How Hierarchical AI Works
Writing in LinkedIn, patent attorney and self-described technology expert David Cain, JD, of Hauptman Ham, LLP, explained that hierarchical AI systems “are structured in layers, each with its own distinct role yet interconnected in a way that forms a cohesive whole. These systems are significant because they mirror the complexity of human decision-making processes, incorporating multiple levels of analysis and action. This multi-tiered approach allows for nuanced problem-solving and decision-making, akin to a seasoned explorer deftly navigating through a multifaceted terrain.”
DeepColony, the researchers wrote, consists of multiple convolutional neural networks (CNNs) that exchange information and cooperate with one another. The system is structured into five levels—labeled 0 through 4—each handling a different part of the analysis:
At level 0, the system determines the number of bacterial colonies and their locations on the plate.
At level 1, the system identifies “good colonies,” meaning those suitable for further identification and analysis.
At level 2, the system assigns each good colony to a bacterial species “based on visual appearance and growth characteristics,” the researchers wrote, referring to the determination as being “pathogen aware, similarity agnostic.”
The CNN used at this stage was trained by using images of 26,213 isolated colonies comprising 32 bacterial species, the researchers wrote in their paper. Most came from clinical laboratories, but some were obtained from the American Type Culture Collection (ATCC), a repository of biological materials and information resources available to researchers.
At level 3, the system attempts to improve accuracy by looking at the larger context of the plate. The goal here is to “determine if observed colonies are similar (pure culture) or different (mixed cultures),” the researchers wrote, describing this step as “similarity aware, pathogen agnostic.” This enables the system to recognize variants of the same strain, the researchers noted, and has the effect of reducing the number of strains identified by the system.
At this level, the system uses two “Siamese CNNs,” which were trained with a dataset of 200,000 image pairs.
Then, at level 4, the system “assesses the clinical significance of the entire plate,” the researchers added. Each plate is labeled as:
“Positive” (significant bacterial growth),
“No significant growth” (negative), or
“Contaminated,” meaning it has three or more “different colony morphologies without a particular pathogen that is prevalent over the others,” the researchers wrote.
If a plate is labeled as “positive,” it can be “further evaluated for possible downstream steps,” using MALDI-TOF mass spectrometry or tests to determine susceptibility to antimicrobial measures, the researchers stated.
“This decision-making process takes into account not only the identification results but also adheres to the specific laboratory guidelines to ensure a proper supportive interpretation in the context of use,” the researchers wrote.
Nearly 100% Agreement with Medical Technologists
To gauge DeepColony’s accuracy, the researchers tested it on a dataset of more than 5,000 urine cultures from a US laboratory. They then compared its analyses with those of human medical technologists who had analyzed the same samples.
Agreement was 99.2% for no-growth cultures, 95.6% for positive cultures, and 77.1% for contaminated or mixed growth cultures, the researchers wrote.
The lower agreement for contaminated cultures was due to “a deliberately precautionary behavior, which is related to ‘safety by design’ criteria,” the researchers noted.
Lead study author Alberto Signoroni, PhD, Associate Professor of Computer Science, University of Brescia, wrote in Nature that many of the plates identified by medical technologists as “contaminated” were labeled as “positive” by DeepColony. “We maximized true negatives while allowing for some false positives, so that DeepColony [can] focus on the most relevant or critical cases,” he said.
Will DeepColony replace medical technologists in clinical laboratories any time soon? Not likely. But the Brescia University study indicates the direction AI in healthcare is headed, with high accuracy and increasing speed. The day may not be far off when pathologists and microbiologists regularly employ AI algorithms to diagnose disease.
Though the new technology could speed diagnoses of cancers and other skin diseases, it would also greatly reduce dermatopathology biopsy referrals and revenue
What effect would elimination of tissue biopsies have on dermatopathology and clinical laboratory revenue? Quite a lot. Dermatologists alone account for a significant portion of skin biopsies sent to dermatopathologists. Thus, any new technology that can “eliminate the need for invasive skin biopsies” would greatly reduce the number of histopathological referrals and reduce revenue to those practices.
“What if we could entirely bypass the biopsy process and perform histology-quality staining without taking tissue and processing tissue in a noninvasive way? Can we create images that diagnosticians can benefit from?” asked Aydogan Ozcan, PhD (above), Chancellor’s Professor of Electrical and Computer Engineering at UCLA’s Samueli School of Engineering, one of the scientists who developed UCLA’s new virtual histology method, during an interview with Medical Device + Diagnostic Industry (MD+DI). (Photo copyright: Nature.)
Could Skin Biopsies be Eliminated?
The UCLA researchers believe their innovative deep learning-enabled imaging framework could possibly circumvent the need for skin biopsies to diagnose skin conditions.
“Here, we present a deep learning-based framework that uses a convolutional neural network to rapidly transform in vivo RCM images of unstained skin into virtually-stained hematoxylin and eosin-like images with microscopic resolution, enabling visualization of the epidermis, dermal-epidermal junction, and superficial dermis layers.
“This application of deep learning-based virtual staining to noninvasive imaging technologies may permit more rapid diagnoses of malignant skin neoplasms and reduce invasive skin biopsies,” the researchers added in their published study.
According to the published study, the UCLA team trained their neural network under an adversarial machine learning scheme to transform grayscale RCM images into virtually stained 3D microscopic images of normal skin, basal cell carcinoma, and pigmented melanocytic nevi. The new images displayed similar morphological features to those shown with the widely used hematoxylin and eosin (H&E) staining method.
“In our studies, the virtually stained images showed similar color contrast and spatial features found in traditionally stained microscopic images of biopsied tissue,” Ozcan told Photonics Media. “This approach may allow diagnosticians to see the overall histological features of intact skin without invasive skin biopsies or the time-consuming work of chemical processing and labeling of tissue.”
The framework covers different skin layers, including the epidermis, dermal-epidermis, and superficial dermis layers. It images deeper into tissue without being invasive and can be quickly performed.
“The virtual stain technology can be streamlined to be almost semi real time,” Ozcan told Medical Device + Diagnostic Industry (MD+DI). “You can have the virtual staining ready when the patient is wrapping up. Basically, it can be within a couple of minutes after you’re done with the entire imaging.”
Currently, medical professionals rely on invasive skin biopsies and histopathological evaluations to diagnose skin diseases and cancers. These diagnostic techniques can result in unnecessary biopsies, scarring, multiple patient visits and increased medical costs for patients, insurers, and the healthcare system.
Improving Time to Diagnosis through Digital Pathology
Another advantage of this virtual technology, the UCLA researchers claim, is that it can provide better images than traditional staining methods, which could improve the ability to diagnose pathological skin conditions and help alleviate human error.
“The majority of the time, small laboratories have a lot of problems with consistency because they don’t use the best equipment to cut, process, and stain tissue,” dermatopathologist Philip Scumpia, MD, PhD, Assistant Professor of Dermatology and Dermatopathology at UCLA Health and one of the authors of the research paper, told MD+DI.
“What ends up happening is we get tissue on a histology slide that’s basically unevenly stained, unevenly put on the microscope, and it gets distorted,” he added, noting that this makes it very hard to make a diagnosis.
Scumpia also added that this new technology would allow digital images to be sent directly to the pathologist, which could reduce processing and laboratory times.
“With electronic medical records now and the ability to do digital photography and digital mole mapping, where you can obtain a whole-body imaging of patients, you could imagine you can also use one of these reflectance confocal devices. And you can take that image from there, add it to the EMR with the virtual histology stain, which will make the images more useful,” Scumpia said. “So now, you can track lesions as they develop.
“What’s really exciting too, is that there’s the potential to combine it with other artificial intelligence, other machine learning techniques that can give more information,” Scumpia added. “Using the reflectance confocal microscope, a clinician who might not be as familiar in dermatopathology could take images and send [them] to a practitioner who could give a more expert diagnosis.”
Faster Diagnoses but Reduced Revenue for Dermatopathologists, Clinical Labs
Ozcan noted that there’s still a lot of work to be done in the clinical assessment, validation, and blind testing of their AI-based staining method. But he hopes the technology can be propelled into a useful tool for clinicians.
“I think this is a proof-of-concept work, and we’re very excited to make it move forward with further advances in technology, in the ways that we acquire 3D information [and] train our neural networks for better and faster virtual staining output,” he told MD+DI.
Though this new technology may reduce the need for invasive biopsies and expedite the diagnosis of skin conditions and cancers—thus improving patient outcomes—what affect might it have on dermatopathology practices?
More research and clinical studies are needed before this new technology becomes part of the diagnosis and treatment processes for skin conditions. Nevertheless, should virtual histology become popular and viable, it could greatly impact the amount of skin biopsy referrals to pathologists, dermatopathologists, and clinical laboratories, thus diminishing a great portion of their revenue.
New study conducted by an international team of researchers suggests that artificial intelligence (AI) may be better than highly-trained humans at detecting certain skin cancers
Artificial intelligence (AI) has been working its way into health technology for several years and, so far, AI tools have been a boon to physicians and health networks. Until now, though, the general view was that it was a supplemental tool for diagnosticians, not a replacement for them. But what if the AI was better at detecting disease than humans, including anatomic pathologists?
Researchers in the Department of Dermatology at Heidelberg University in Germany have concluded that AI can be more accurate at identifying certain cancers. The challenge they designed for their study involved skin biopsies and dermatologists.
They pitted a deep-learning convolutional neural network (CNN) against 58 dermatologists from 17 countries to determine which was more accurate at detecting malignant melanomas—humans or AI. A CNN is an artificial network based on the biological processes that occur when neurons in the brain are connected to each other and respond to what the eye sees.
The CNN won.
“For the first time we compared a CNN’s diagnostic performance with a large international group of 58 dermatologists, including 30 experts. Most dermatologists were outperformed by the CNN. Irrespective of any physicians’ experience, they may benefit from assistance by a CNN’s image classification,” the report noted.
“I expected only a performance on an even level with the physicians. The outperformance even of the average experienced and trained dermatologists was a major surprise,” Holger Haenssle, PhD, Professor of Dermatology at Heidelberg University and one of the authors of the study, told Healthline. Anatomic pathologists will want to follow the further development of this research and its associated diagnostic technologies. (Photo copyright: University of Heidelberg.)
Does AI Tech Have Superior Visual Acuity Compared to Human Eyes?
The dermatologists who participated in the study had varying degrees of experience in dermoscopy, also known as dermatoscopy. Thirty of the doctors had more than five-year’s experience and were considered to be expert level. Eleven of the dermatologists were considered “skilled” with two- to five-year’s experience. The remaining 17 doctors were termed beginners with less than two-year’s experience.
To perform the study, the researchers first compiled a set of 100 dermoscopic images that showed melanomas and benign moles called Nevi. Dermoscopes (or dermatoscopes) create images using a magnifying glass and light source pressed against the skin. The resulting magnified, high-resolution images allow for easier, more accurate diagnoses than inspection with the naked eye.
During the first stage of the research, the dermatologists were asked to diagnose whether a lesion was melanoma or benign by looking at the images with their naked eyes. They also were asked to render their opinions for any needed action, such as surgery and follow-up care based on their diagnoses.
After this part of the study, the dermatologists on average identified 86.6% of the melanomas and 71.3% of the benign moles. More experienced doctors identified the melanomas at 89%, which was slightly higher than the average of the group.
The researchers also showed 300 images of malignant and benign skin lesions to the CNN. The AI accurately identified 95% of the melanomas by analyzing the images.
“The CNN missed fewer melanomas, meaning it had a higher sensitivity than the dermatologists, and it misdiagnosed fewer benign moles as malignant melanoma, which means it had a higher specificity. This would result in less unnecessary surgery,” Haenssle told CBS News.
In a later part of the research, the dermatologists were shown the images a second time and provided clinical information about the patients, including age, gender, and location of the lesion. They were again instructed to make diagnoses and projected care decisions. With the additional information, the doctors’ average detection of melanomas increased to 88.9% and their recognition of benign moles increased to 75.7%. Still below the results of the CNN.
These findings suggest that the visual pattern recognition of AI technology could be a meaningful tool to help physicians and researchers diagnose certain cancers.
“In the future, I think AI will be integrated into practice as a diagnostic aide, particularly in primary care, to support the decision to excise a lesion, refer, or otherwise to reassure that it is benign,” Victoria Mar, PhD, an Adjunct Senior Lecturer in the Department of Public Health and Preventative Medicine at Australia’s Monash University, told Healthline.
“There is the potential for AI technology to be integrated with 2D or 3D skin imaging systems, which means that the majority of benign lesions would be already filtered by the machine, so that we can spend more time concentrating on the difficult or more concerning lesions,” she said. “To me, this means a more productive interaction with the patient, where we can focus on appropriate management and provide more streamlined care.”
AI Performs Well in Other Studies Involving Skin Biopsies
This study is not the only research that suggests entities besides humans may be utilized in diagnosing some cancers from images. Last year, computer scientists at Stanford University performed similar research and found comparable results. For that study, the researchers created and trained an algorithm to visually diagnose potential skin cancers by looking at a database of skin images. They then showed photos of skin lesions to 21 dermatologists and asked for their diagnoses based on the images. They found the accuracy of their AI matched the performance of the doctors when diagnosing skin cancer from viewed images.
While many dermatologists read patient biopsies on their own, they also refer high volumes of skin biopsies to anatomic pathologists. A technology that can accurately diagnose skin cancers could potentially impact the workload received by clinical laboratories and anatomic pathology groups.
