Researchers intend their new AI image retrieval tool to help pathologists locate similar case images to reference for diagnostics, research, and education
Researchers at Stanford University turned to an unusual source—the X social media platform (formerly known as Twitter)—to train an artificial intelligence (AI) system that can look at clinical laboratory pathology images and then retrieve similar images from a database. This is an indication that pathologists are increasingly collecting and storing images of representative cases in their social media accounts. They then consult those libraries when working on new cases that have unusual or unfamiliar features.
The Stanford Medicine scientists trained their AI system—known as Pathology Language and Image Pretraining (PLIP)—on the OpenPath pathology dataset, which contains more than 200,000 images paired with natural language descriptions. The researchers collected most of the data by retrieving tweets in which pathologists posted images accompanied by comments.
“It might be surprising to some folks that there is actually a lot of high-quality medical knowledge that is shared on Twitter,” said researcher James Zou, PhD, Assistant Professor of Biomedical Data Science and senior author of the study, in a Stanford Medicine SCOPE blog post, which added that “the social media platform has become a popular forum for pathologists to share interesting images—so much so that the community has widely adopted a set of 32 hashtags to identify subspecialties.”
“It’s a very active community, which is why we were able to curate hundreds of thousands of these high-quality pathology discussions from Twitter,” Zou said.
“The main application is to help human pathologists look for similar cases to reference,” James Zou, PhD (above), Assistant Professor of Biomedical Data Science, senior author of the study, and his colleagues wrote in Nature Medicine. “Our approach demonstrates that publicly shared medical information is a tremendous resource that can be harnessed to develop medical artificial intelligence for enhancing diagnosis, knowledge sharing, and education.” Leveraging pathologists’ use of social media to store case images for future reference has worked out well for the Stanford Medicine study. (Photo copyright: Stanford University.)
Retrieving Pathology Images from Tweets
“The lack of annotated publicly-available medical images is a major barrier for innovations,” the researchers wrote in Nature Medicine. “At the same time, many de-identified images and much knowledge are shared by clinicians on public forums such as medical Twitter.”
In this case, the goal “is to train a model that can understand both the visual image and the text description,” Zou said in the SCOPE blog post.
“Pathology is perhaps even more suited to Twitter than many other medical fields because for most pathologists, the bulk of our daily work revolves around the interpretation of images for the diagnosis of human disease,” wrote Jerad M. Gardner, MD, a dermatopathologist and section head of bone/soft tissue pathology at Geisinger Medical Center in Danville, Pa., in a blog post about the Pathology Hashtag Ontology project. “Twitter allows us to easily share images of amazing cases with one another, and we can also discuss new controversies, share links to the most cutting edge literature, and interact with and promote the cause of our pathology professional organizations.”
The researchers used the 32 subspecialty hashtags to retrieve English-language tweets posted from 2006 to 2022. Images in the tweets were “typically high-resolution views of cells or tissues stained with dye,” according to the SCOPE blog post.
The researchers collected a total of 232,067 tweets and 243,375 image-text pairs across the 32 subspecialties, they reported. They augmented this with 88,250 replies that received the highest number of likes and had at least one keyword from the ICD-11 codebook. The SCOPE blog post noted that the rankings by “likes” enabled the researchers to screen for high-quality replies.
They then refined the dataset by removing duplicates, retweets, non-pathology images, and tweets marked by Twitter as being “sensitive.” They also removed tweets containing question marks, as this was an indicator that the practitioner was asking a question about an image rather than providing a description, the researchers wrote in Nature Medicine.
They cleaned the text by removing hashtags, Twitter handles, HTML tags, emojis, and links to websites, the researchers noted.
The final OpenPath dataset included:
116,504 image-text pairs from Twitter posts,
59,869 from replies, and
32,041 image-text pairs scraped from the internet or obtained from the LAION dataset.
The latter is an open-source database from Germany that can be used to train text-to-image AI software such as Stable Diffusion.
Training the PLIP AI Platform
Once they had the dataset, the next step was to train the PLIP AI model. This required a technique known as contrastive learning, the researchers wrote, in which the AI learns to associate features from the images with portions of the text.
As explained in Baeldung, an online technology publication, contrastive learning is based on the idea that “it is easier for someone with no prior knowledge, like a kid, to learn new things by contrasting between similar and dissimilar things instead of learning to recognize them one by one.”
“The power of such a model is that we don’t tell it specifically what features to look for. It’s learning the relevant features by itself,” Zou said in the SCOPE blog post.
The resulting AI PLIP tool will enable “a clinician to input a new image or text description to search for similar annotated images in the database—a sort of Google Image search customized for pathologists,” SCOPE explained.
“Maybe a pathologist is looking at something that’s a bit unusual or ambiguous,” Zou told SCOPE. “They could use PLIP to retrieve similar images, then reference those cases to help them make their diagnoses.”
The Stanford University researchers continue to collect pathology images from X. “The more data you have, the more it will improve,” Zou said.
Pathologists will want to keep an eye on the Stanford Medicine research team’s progress. The PLIP AI tool may be a boon to diagnostics and improve patient outcomes and care.
Though the new technology could speed diagnoses of cancers and other skin diseases, it would also greatly reduce dermatopathology biopsy referrals and revenue
What effect would elimination of tissue biopsies have on dermatopathology and clinical laboratory revenue? Quite a lot. Dermatologists alone account for a significant portion of skin biopsies sent to dermatopathologists. Thus, any new technology that can “eliminate the need for invasive skin biopsies” would greatly reduce the number of histopathological referrals and reduce revenue to those practices.
“What if we could entirely bypass the biopsy process and perform histology-quality staining without taking tissue and processing tissue in a noninvasive way? Can we create images that diagnosticians can benefit from?” asked Aydogan Ozcan, PhD (above), Chancellor’s Professor of Electrical and Computer Engineering at UCLA’s Samueli School of Engineering, one of the scientists who developed UCLA’s new virtual histology method, during an interview with Medical Device + Diagnostic Industry (MD+DI). (Photo copyright: Nature.)
Could Skin Biopsies be Eliminated?
The UCLA researchers believe their innovative deep learning-enabled imaging framework could possibly circumvent the need for skin biopsies to diagnose skin conditions.
“Here, we present a deep learning-based framework that uses a convolutional neural network to rapidly transform in vivo RCM images of unstained skin into virtually-stained hematoxylin and eosin-like images with microscopic resolution, enabling visualization of the epidermis, dermal-epidermal junction, and superficial dermis layers.
“This application of deep learning-based virtual staining to noninvasive imaging technologies may permit more rapid diagnoses of malignant skin neoplasms and reduce invasive skin biopsies,” the researchers added in their published study.
According to the published study, the UCLA team trained their neural network under an adversarial machine learning scheme to transform grayscale RCM images into virtually stained 3D microscopic images of normal skin, basal cell carcinoma, and pigmented melanocytic nevi. The new images displayed similar morphological features to those shown with the widely used hematoxylin and eosin (H&E) staining method.
“In our studies, the virtually stained images showed similar color contrast and spatial features found in traditionally stained microscopic images of biopsied tissue,” Ozcan told Photonics Media. “This approach may allow diagnosticians to see the overall histological features of intact skin without invasive skin biopsies or the time-consuming work of chemical processing and labeling of tissue.”
The framework covers different skin layers, including the epidermis, dermal-epidermis, and superficial dermis layers. It images deeper into tissue without being invasive and can be quickly performed.
“The virtual stain technology can be streamlined to be almost semi real time,” Ozcan told Medical Device + Diagnostic Industry (MD+DI). “You can have the virtual staining ready when the patient is wrapping up. Basically, it can be within a couple of minutes after you’re done with the entire imaging.”
Currently, medical professionals rely on invasive skin biopsies and histopathological evaluations to diagnose skin diseases and cancers. These diagnostic techniques can result in unnecessary biopsies, scarring, multiple patient visits and increased medical costs for patients, insurers, and the healthcare system.
Improving Time to Diagnosis through Digital Pathology
Another advantage of this virtual technology, the UCLA researchers claim, is that it can provide better images than traditional staining methods, which could improve the ability to diagnose pathological skin conditions and help alleviate human error.
“The majority of the time, small laboratories have a lot of problems with consistency because they don’t use the best equipment to cut, process, and stain tissue,” dermatopathologist Philip Scumpia, MD, PhD, Assistant Professor of Dermatology and Dermatopathology at UCLA Health and one of the authors of the research paper, told MD+DI.
“What ends up happening is we get tissue on a histology slide that’s basically unevenly stained, unevenly put on the microscope, and it gets distorted,” he added, noting that this makes it very hard to make a diagnosis.
Scumpia also added that this new technology would allow digital images to be sent directly to the pathologist, which could reduce processing and laboratory times.
“With electronic medical records now and the ability to do digital photography and digital mole mapping, where you can obtain a whole-body imaging of patients, you could imagine you can also use one of these reflectance confocal devices. And you can take that image from there, add it to the EMR with the virtual histology stain, which will make the images more useful,” Scumpia said. “So now, you can track lesions as they develop.
“What’s really exciting too, is that there’s the potential to combine it with other artificial intelligence, other machine learning techniques that can give more information,” Scumpia added. “Using the reflectance confocal microscope, a clinician who might not be as familiar in dermatopathology could take images and send [them] to a practitioner who could give a more expert diagnosis.”
Faster Diagnoses but Reduced Revenue for Dermatopathologists, Clinical Labs
Ozcan noted that there’s still a lot of work to be done in the clinical assessment, validation, and blind testing of their AI-based staining method. But he hopes the technology can be propelled into a useful tool for clinicians.
“I think this is a proof-of-concept work, and we’re very excited to make it move forward with further advances in technology, in the ways that we acquire 3D information [and] train our neural networks for better and faster virtual staining output,” he told MD+DI.
Though this new technology may reduce the need for invasive biopsies and expedite the diagnosis of skin conditions and cancers—thus improving patient outcomes—what affect might it have on dermatopathology practices?
More research and clinical studies are needed before this new technology becomes part of the diagnosis and treatment processes for skin conditions. Nevertheless, should virtual histology become popular and viable, it could greatly impact the amount of skin biopsy referrals to pathologists, dermatopathologists, and clinical laboratories, thus diminishing a great portion of their revenue.
Computer-aided diagnostic system combines optical dermatoscopy, spectrophotometry and high-frequency ultrasound imaging techniques to differentiate malignant lesions from benign moles
Detecting skin cancer via the use of skin biopsies is the bread and butter of many dermatopathology practices. But new technologies that can instantly detect and distinguish different types of skin malignancies may result in a reduced flow of skin biopsies to dermatopathologists in the not-too-distant future.
The new technique achieved a more accurate method of differentiating melanoma from benign lesions, according to the researchers.
“The novelty of our method is that it combines diagnostic information obtained from different non-invasive imaging technologies such as optical spectrophotometry and [high-frequency] ultrasound. Based on the results of our research, we can confirm that the developed automated system can complement the non-invasive diagnostic methods currently applied in the medical practice by efficiently differentiating melanoma from a melanocytic mole,” said Renaldas Raišutis, PhD, coauthor of the study, in a KTU news release.
According to the Skin Cancer Foundation, one in five Americans will develop skin cancer by the age of 70 and more than two people in the United States die of skin cancer every hour. Early diagnosis is vital. If a malignant melanoma—the most lethal type of skin cancer—is found early, the five-year survival rate is 99%. (Graphic copyright: Healthline.)
“An efficient diagnosis of an early-stage malignant skin tumor could save critical time, more patients could be examined, and more of them could be saved,” Raišutis said in the news release. He added that the CADx-based diagnostic system is aimed at medical professionals but at a price that makes it affordable for smaller medical institutions. The Lithuanian team also is working to design a system that could be marketed for home use.
New Non-invasive Optical Technology May Reduce Demand for Skin Biopsies
A systematic review article published in Frontiers in Medicine Dermatology compared current diagnostic techniques for melanoma. It noted, “The current gold standard for melanoma diagnosis is the administration of dermoscopy, followed by a biopsy and subsequent histopathological analysis of the excised tissue. To minimize the risk of misdiagnosis of true melanomas, a significant number of dermoscopically ambiguous lesions are biopsied [increasing] the overall diagnostic costs and time to obtain the final diagnosis.”
But continuing technological innovations may be setting the stage for a reduction in the number of skin biopsies performed each year. In addition to the novel diagnostic method announced by the Lithuanian researchers, an Israeli scientist has created an innovative optical technology that can instantly and non-invasively detect and distinguish between three primary skin cancers:
Tel Aviv University Physics Professor Abraham Katzir, PhD (above), demonstrates his new method of detecting cancerous skin lesion, which employs infrared sensors and optical fibers to determine the properties of various lesions on the skin and identify them based on their coloration within the infrared spectrum. (Photo copyright: Tel Aviv University.)
“We figured that with the help of devices that can identify these colors, healthy skin and each of the benign and malignant lesions would have different ‘colors,’ which would enable us to identify melanoma,” Katzir said in an Israel 21c news article.
“Melanoma is a life-threatening cancer, so it is very important to diagnose it early on, when it is still superficial,” Katzir told Israel 21c, adding that the new technology has the potential to cause “dramatic change” in the field of diagnosing and treating skin cancer, “and perhaps other types of cancer as well.”
As advancements in the non-invasive diagnosis of skin cancers continue, dermatopathologists—and in fact all anatomic and histopathology practices—should prepare for the financial impact this change may have on their clinical practices as demand for skin biopsies decreases.
Team of bioengineers succeeds in putting three different imaging technologies into a handheld probe that could be used by physicians to assess skin lesions in their offices
Dermatopathologists and pathology practice administrators will be keenly interested in a new, hand-held diagnostic device that is designed to reduce the need for skin biopsies. Because of high volume of skin biopsies referred to pathologists, any significant reduction in the number of such case referrals would have negative revenue impact on medical laboratories that process and diagnose these specimens.
This innovative work was done at the University of Texas at Austin’s Cockrell School of Engineering. The research team developed a probe that uses three different light modalities to detect melanoma and other skin cancer lesions in real-time, according to a news release. (more…)
Company intends to use pattern recognition software to evaluate risk of skin cancer
A “do it yourself” dermatopathology service for consumers is coming soon, according to Health Discovery Corporation (HDC) of Savannah, Georgia. The company is preparing to introduce a cell phone-based tool to help consumers recognize whether a mole or other skin lesion needs examination by a dermatologist.
Using their cell phone cameras, consumers would click a photo of the skin lesion, then forward that image to a computer at HDC. Using pattern recognition algorithms developed by the company, called Support Vector Machines, the computer would analyze the image. A report telling the consumer whether the lesion was low, medium or high risk for skin cancer would be sent as a text message. This text message would include a list of dermatologists located near the consumer. The list of dermatologist referrals would be targeted to the user’s geographic area. HPC would compile this list, based on GPS data collected from the cell phone transmission.
