News, Analysis, Trends, Management Innovations for
Clinical Laboratories and Pathology Groups

Hosted by Robert Michel

News, Analysis, Trends, Management Innovations for
Clinical Laboratories and Pathology Groups

Hosted by Robert Michel
Sign In

Stanford Researchers Use Text and Images from Pathologists’ Twitter Accounts to Train New Pathology AI Model

Researchers intend their new AI image retrieval tool to help pathologists locate similar case images to reference for diagnostics, research, and education

Researchers at Stanford University turned to an unusual source—the X social media platform (formerly known as Twitter)—to train an artificial intelligence (AI) system that can look at clinical laboratory pathology images and then retrieve similar images from a database. This is an indication that pathologists are increasingly collecting and storing images of representative cases in their social media accounts. They then consult those libraries when working on new cases that have unusual or unfamiliar features.

The Stanford Medicine scientists trained their AI system—known as Pathology Language and Image Pretraining (PLIP)—on the OpenPath pathology dataset, which contains more than 200,000 images paired with natural language descriptions. The researchers collected most of the data by retrieving tweets in which pathologists posted images accompanied by comments.

“It might be surprising to some folks that there is actually a lot of high-quality medical knowledge that is shared on Twitter,” said researcher James Zou, PhD, Assistant Professor of Biomedical Data Science and senior author of the study, in a Stanford Medicine SCOPE blog post, which added that “the social media platform has become a popular forum for pathologists to share interesting images—so much so that the community has widely adopted a set of 32 hashtags to identify subspecialties.”

“It’s a very active community, which is why we were able to curate hundreds of thousands of these high-quality pathology discussions from Twitter,” Zou said.

The Stanford researchers published their findings in the journal Nature Medicine titled, “A Visual-Language Foundation Model for Pathology Image Analysis Using Medical Twitter.”

James Zou, PhD

“The main application is to help human pathologists look for similar cases to reference,” James Zou, PhD (above), Assistant Professor of Biomedical Data Science, senior author of the study, and his colleagues wrote in Nature Medicine. “Our approach demonstrates that publicly shared medical information is a tremendous resource that can be harnessed to develop medical artificial intelligence for enhancing diagnosis, knowledge sharing, and education.” Leveraging pathologists’ use of social media to store case images for future reference has worked out well for the Stanford Medicine study. (Photo copyright: Stanford University.)

Retrieving Pathology Images from Tweets

“The lack of annotated publicly-available medical images is a major barrier for innovations,” the researchers wrote in Nature Medicine. “At the same time, many de-identified images and much knowledge are shared by clinicians on public forums such as medical Twitter.”

In this case, the goal “is to train a model that can understand both the visual image and the text description,” Zou said in the SCOPE blog post.

Because X is popular among pathologists, the United States and Canadian Academy of Pathology (USCAP), and Pathology Hashtag Ontology project, have recommended a standard series of hashtags, including 32 hashtags for subspecialties, the study authors noted.

Examples include:

“Pathology is perhaps even more suited to Twitter than many other medical fields because for most pathologists, the bulk of our daily work revolves around the interpretation of images for the diagnosis of human disease,” wrote Jerad M. Gardner, MD, a dermatopathologist and section head of bone/soft tissue pathology at Geisinger Medical Center in Danville, Pa., in a blog post about the Pathology Hashtag Ontology project. “Twitter allows us to easily share images of amazing cases with one another, and we can also discuss new controversies, share links to the most cutting edge literature, and interact with and promote the cause of our pathology professional organizations.”

The researchers used the 32 subspecialty hashtags to retrieve English-language tweets posted from 2006 to 2022. Images in the tweets were “typically high-resolution views of cells or tissues stained with dye,” according to the SCOPE blog post.

The researchers collected a total of 232,067 tweets and 243,375 image-text pairs across the 32 subspecialties, they reported. They augmented this with 88,250 replies that received the highest number of likes and had at least one keyword from the ICD-11 codebook. The SCOPE blog post noted that the rankings by “likes” enabled the researchers to screen for high-quality replies.

They then refined the dataset by removing duplicates, retweets, non-pathology images, and tweets marked by Twitter as being “sensitive.” They also removed tweets containing question marks, as this was an indicator that the practitioner was asking a question about an image rather than providing a description, the researchers wrote in Nature Medicine.

They cleaned the text by removing hashtags, Twitter handles, HTML tags, emojis, and links to websites, the researchers noted.

The final OpenPath dataset included:

  • 116,504 image-text pairs from Twitter posts,
  • 59,869 from replies, and
  • 32,041 image-text pairs scraped from the internet or obtained from the LAION dataset.

The latter is an open-source database from Germany that can be used to train text-to-image AI software such as Stable Diffusion.

Training the PLIP AI Platform

Once they had the dataset, the next step was to train the PLIP AI model. This required a technique known as contrastive learning, the researchers wrote, in which the AI learns to associate features from the images with portions of the text.

As explained in Baeldung, an online technology publication, contrastive learning is based on the idea that “it is easier for someone with no prior knowledge, like a kid, to learn new things by contrasting between similar and dissimilar things instead of learning to recognize them one by one.”

“The power of such a model is that we don’t tell it specifically what features to look for. It’s learning the relevant features by itself,” Zou said in the SCOPE blog post.

The resulting AI PLIP tool will enable “a clinician to input a new image or text description to search for similar annotated images in the database—a sort of Google Image search customized for pathologists,” SCOPE explained.

“Maybe a pathologist is looking at something that’s a bit unusual or ambiguous,” Zou told SCOPE. “They could use PLIP to retrieve similar images, then reference those cases to help them make their diagnoses.”

The Stanford University researchers continue to collect pathology images from X. “The more data you have, the more it will improve,” Zou said.

Pathologists will want to keep an eye on the Stanford Medicine research team’s progress. The PLIP AI tool may be a boon to diagnostics and improve patient outcomes and care.

—Stephen Beale

Related Information:

New AI Tool for Pathologists Trained by Twitter (Now Known as X)

A Visual-Language Foundation Model for Pathology Image Analysis Using Medical Twitter

AI + Twitter = Foundation Visual-Language AI for Pathology

Pathology Foundation Model Leverages Medical Twitter Images, Comments

A Visual-Language Foundation Model for Pathology Image Analysis Using Medical Twitter (Preprint)

Pathology Language and Image Pre-Training (PLIP)

Introducing the Pathology Hashtag Ontology

Doctors in India Sound Alarm: CRE Infections are Becoming Common in India and Killing Two-Thirds of Patients Who Contract Them While Undergoing Cancer Treatment!

As infectious bacteria become even more resistant to antibiotics, chronic disease patients with weakened immune systems are in particular danger

Microbiologists and clinical laboratory managers in the United States may find it useful to learn that exceptionally virulent strains of bacteria are causing increasing numbers of cancer patient deaths in India. Given the speed with which infectious diseases spread throughout the world, it’s not surprising that deaths due to similar hospital-acquired infections (HAIs) are increasing in the US as well.

Recent news reporting indicates that an ever-growing number of cancer patients in the world’s second most populous nation are struggling to survive these infections while undergoing chemotherapy and other treatments for their cancers.

In some ways, this situation is the result of more powerful antibiotics. Today’s modern antibiotics help physicians, pathologists, and clinical laboratories protect patients from infectious disease. However, it’s a tragic fact that those same powerful drugs are making patients with chronic diseases, such as cancer, more susceptible to death from HAIs caused by bacteria that are becoming increasingly resistant to those same antibiotics.

India is a prime example of that devastating dichotomy. Bloomberg reported that a study conducted by Abdul Ghafur, MD, an infectious disease physician with Apollo Hospitals in Chennai, India, et al, concluded that “Almost two-thirds of cancer patients with a carbapenem-resistant infection are dead within four weeks, vs. a 28-day mortality rate of 38% in patients whose infections are curable.”

This news should serve as an alert to pathologists, microbiologists, and clinical laboratory leaders in the US as these same superbugs—which resist not only antibiotics but other drugs as well—may become more prevalent in this country.

 ‘We Don’t Know What to Do’

The dire challenge facing India’s cancer patients is due to escalating bloodstream infections associated with carbapenem-resistant enterobacteriaceae (CRE), a particularly deadly bacteria that has become resistant to even the most potent carbapenem antibiotics, generally considered drugs of last resort for dealing with life-threatening infections.

Lately, the problem has only escalated. “We are facing a difficult scenario—to give chemotherapy and cure the cancer and get a drug-resistant infection and the patient dying of infections.” Ghafur told Bloomberg. “We don’t know what to do. The world doesn’t know what to do in this scenario.”

Ghafur added, “However wonderful the developments in the field of oncology, they are not going to be useful, because we know cancer patients die of infections.”

Abdul Ghafur, MD (above), an infectious disease physician with Apollo Hospitals in Chennai, India, told The Better India that, “Indians, are obsessed with antibiotics and believe that they can cure almost all infections, including viral infections! Moreover, at least half of the prescriptions by Indian doctors include an antibiotic. Sadly, the public believes that whenever we get cold and cough, we need to swallow antibiotics for three days along with paracetamol [acetaminophen]! This is a myth that urgently needs to disappear!” (Photo copyright: Longitude Prize.)

The problem in India, Bloomberg reports, is exacerbated by contaminated food and water. “Germs acquired through ingesting contaminated food and water become part of the normal gut microbiome, but they can turn deadly if they escape the bowel and infect the urinary tract, blood, and other tissues.” And chemotherapy patients, who likely have weakened digestive tracts, suffer most when the deadly germs reach the urinary tract, blood, and surrounding tissues.

“Ten years ago, carbapenem-resistant superbug infections were rare. Now, infections such as carbapenem-resistant klebsiella bloodstream infection, urinary infection, pneumonia, and surgical site infections are a day-to-day problem in our (Indian) hospitals. Even healthy adults in the community may carry these bacteria in their gut in Indian metropolitan cities; up to 5% of people carry these superbugs in their intestines,” Ghafur told The Better India.

What are CRE and Why Are They So Deadly?

CRE are part of the enterobacteriaceae bacterial family, which also includes Escherichia coli (E. coli) and Klebsiella pneumoniae. CRE, according to the Centers for Disease Control and Prevention (CDC), are considered “antibiotic-resistant” because antibiotic agents known as carbapenems are becoming increasing less effective at treating enterobacteriaceae.

In fact, a 2018 study conducted by the All India Institute of Medical Sciences (AIIMS) in New Delhi, which was published in the Journal of Global Infectious Diseases (JGID), found that bloodstream infections due to CRE were the “leading cause” of illness and death in patients with hematological malignancies, such as leukemia.

“These patients receive chemotherapy during treatment, which lead to severe mucositis of gastrointestinal tract and myelosuppression. It was hypothesized that the gut colonizer translocate into blood circulation causing [bloodstream infection],” the AIIMS paper states.

US Cases of C. auris Also Linked to CRE

Deaths in the US involving the fungus Candida auris (C. auris) have been linked to CRE as well. And, people who were hospitalized outside the US may be at particular risk.

The CDC reported on a Maryland resident who was hospitalized in Kenya with a carbapenemase-producing infection, which was later diagnosed as C. auris. The CDC describes C. auris as “an emerging drug-resistant yeast of high public concern … C auris frequently co-occurs with carbapenemase-producing organisms like CRE.”

The graphic above, developed by the NYT from CDC data, shows that Candida auris is found globally and not restricted to poor or resource-strapped nations. “The fungus seems to have emerged in several locations at once, not from a single source,” the NYT reports. This means clinical laboratories can expect to be processing more tests to identify the deadly fungus. (Graphic copyright: New York Times/CDC.)

Drug-resistant germs are a public health threat that has grown beyond overuse of antibiotics to an “explosion of resistant fungi,” reported the New York Times (NYT).

“It’s an enormous problem. We depend on being able to treat those patients with antifungals,” Matthew Fisher, PhD, Professor of Fungal Disease Epidemiology at Imperial College London, told the NYT

The NYT article states that “Nearly half of patients who contract C. auris die within 90 days, according to the CDC. Yet the world’s experts have not nailed down where it came from in the first place.”

Cases of C. auris in the US are showing up in New York, New Jersey, and Illinois and is arriving on travelers from many countries, including India, Pakistan, South Africa, Spain, United Kingdom, and Venezuela.  

“It is a creature from the black lagoon,” Tom Chiller, MD, Chief of the Mycotic Diseases Branch at the CDC told the NYT. “It bubbled up and now it is everywhere.”

Since antibiotics are used heavily in agriculture and farming worldwide, the numbers of antibiotic-resistant infections will likely increase. Things may get worse, before they get better.

Pathologists, microbiologists, oncologists, and clinical laboratories involved in caring for patients with antibiotic-resistant infections will want to fully understand the dangers involved, not just to patients, but to healthcare workers as well.

—Donna Marie Pocius

Related Information:

Superbugs Deadlier than Cancer Put Chemotherapy into Question

Taking Antibiotics for a Viral Infection? A Doc Shares Why You Should Think Twice

Healthcare-Associated Infections: CRE

Rectal Carriage of Carbapenem-resistant enterobacteriaceae: A Menace to Highly Vulnerable Patients

Clinical Study of Carbapenem Sensitive and Resistant Gram-negative Bacteria in Neutropenic and Nonneutropenic Patients: The First Series from India

Candida Auris in a U.S. Patient with Carbapenemase-Producing Organisms and Recent Hospitalization in Kenya

Deadly Germs, Lost Cures: A Mysterious Infection, Spanning the Globe in a Climate of Secrecy

University of Edinburgh Study Finds Antimicrobial Bacteria in Hospital Wastewater in Research That Has Implications for Microbiologists

Pathologists and Clinical Laboratories to Play Critical Role in Developing New Tools to Fight Antibiotic Resistance

Lurking Below: NIH Study Reveals Surprising New Source of Antibiotic Resistance That Will Interest Microbiologists and Medical Laboratory Scientists

Comparison of In Vitro Diagnostic Industry’s Top Five Trends for 2015 and 2016 Reveals Rapid Technology Advances Intended to Give Clinical Laboratories New Diagnostic Tools

Of the five trends described in a report published by Kalorama, only two made the list for both years: Consolidation within the IVD industry and growth in molecular point of care

What a difference one year can make in the most significant trends influencing the in vitro diagnostics (IVD) industry, which also influences clinical laboratories, the largest customers of IVD manufacturers. These insights come from comparing the top five IVD trends for 2016 as identified by Kalorama Information from its top five IVD trends that it says dominated during 2015.

Kalorama is a division of MarketResearch.com, a company that publishes market research in the life sciences. In a report titled, “Five IVD Market Trends to Watch for in 2016,” it published its picks for the top five trends in IVD testing for 2016. The five most prominent trends recognized by the healthcare research marketer are as follows: (more…)

Partnership of Illumina and bioMérieux Proposes an Epidemiology Service to Provide Hospitals and Public Health Officials ‘Out-of-the-Box’ Genomic Pathogen Solutions

This collaborative effort with microbiology labs will keep microbiologists at the forefront of infectious disease diagnostics

A partnership of San Diego-based genome sequencing company Illumina, and the French multinational, in vitro diagnostics company bioMérieux, plans to launch a next-generation sequencing (NGS) epidemiology service that will allow microbiologists to rapidly identify strains that threaten hospital inpatients and public health, according to a press release distributed by the Illumina-bioMérieux team.

Illumina-bioMérieux Service to Aid Hospital and Public Health Labs 

Illumina designated sequencing laboratories with Illumina MiSeq® systems will collaborate with microbiologists working in hospital and public health laboratories to prevent, rapidly track, and contain infectious disease agents in hospitals and communities. (more…)

Two Different Point-of-Care Test Devices for Malaria Show Why Emerging Technologies Can Be Disruptive to Clinical Pathology Laboratories

Separate research projects at University of Washington and in the United Kingdom are producing handheld diagnostic devices to accurately detect Malaria

Two new handheld, point-of-care test (POC) devices for malaria  could save millions of lives in third-world countries. At the same time, these POC devices may lead to inexpensive alternatives for diagnosing common diseases in developed nations as well.

Clinical laboratory test developers see a big opportunity in developing assays to detect Malaria. That is because an estimated 200 million cases of malaria are diagnosed annually, resulting in the death of about 100 million people each year.

Recently, two organizations released news about the specific testing devices they have developed to detect malaria. One group is at the University of Washington in Seattle, Washington. The other group is NanoMal, a biotechnology company located in the United Kingdom. (more…)

;