News, Analysis, Trends, Management Innovations for
Clinical Laboratories and Pathology Groups

Hosted by Robert Michel

News, Analysis, Trends, Management Innovations for
Clinical Laboratories and Pathology Groups

Hosted by Robert Michel
Sign In

University College London Study Shows Direct-to-Consumer DNA Tests Not Reliable in Assessing Disease Risk

Regulatory agencies in UK and US have yet to address dangers inherent in customer misunderstanding of DTC medical laboratory genetic test results

Direct-to-consumer (DTC) medical laboratory genetic tests are gaining popularity across the globe. But recent research out of the United Kingdom questions the reliability of these tests. The study, according to The Guardian, found that “Over the counter genetic tests in the UK that assess the risk of cancer or heart problems fail to identify 89% of those in danger of getting killer diseases.”

Researchers at University College London (UCL) examined 926 polygenic scores for risk of 310 different diseases from details retrieved from the Polygenic Score (PGS) Catalog. This catalog is an open-access database of published polygenic scores.

According the PGS website, “each PGS in the catalog is consistently annotated with relevant metadata; including scoring files (variants, effect alleles/weights), annotations of how the PGS was developed and applied, and evaluations of their predictive performance.”

However, the researchers told The Guardian, “Polygenic risk scores performed poorly in population screening, individual risk prediction, and population risk stratification. Strong claims about the effect of polygenic risk scores on healthcare seem to be disproportionate to their performance.”

The UCL researchers published their findings in the journal BMJ Medicine titled, “Performance of Polygenic Risk Scores in Screening, Prediction, and Risk Stratification: Secondary Analysis of Data in the Polygenic Source Catalog.”

“Strong claims have been made about the potential of polygenic risk scores in medicine, but our study shows that this is not justified,” Aroon Hingorani, PhD (above), Professor of Genetic Epidemiology at UCL and lead author of the study, told The Guardian. “We found that, when held to the same standards as employed for other tests in medicine, polygenic risk scores performed poorly for prediction and screening across a range of common diseases.” Consumer misunderstanding of DTC medical laboratory genetic tests is a real danger. (Photo copyright: University College London.)

Polygenic Scores Not Beneficial to Cancer Screening

To complete their study, the UCL researchers compared PGS genetic risk data to conventional clinical laboratory testing methods and discovered some troubling results. They include:

  • On average, only 11% of individuals who developed a disease had been identified by the tests.
  • A 5% false positive rate where people were informed that they would get a disease within 10 years but did not.
  • PGS only identified 10% of people who later developed breast cancer.
  • PGS only identified 12% of individuals who later developed coronary artery disease.

The researchers state in their BMJ Medicine paper that polygenic risk scores are not the same as testing for certain gene mutations, which could be critical in screening for some cancers. They also wrote that discovering genetic variants associated with the risk for disease is still crucial for drug development.

“It has been suggested that polygenic risk scores could be introduced early on to help prevent breast cancer and heart disease but, in the examples we looked at, we found that the scores contributed little, if any, health benefit while adding cost and complexity,” research physician and epidemiologist Sir Nicholas Wald, FRS, FRCP, FMedSci, Professor of Preventive Medicine at UCL Institute of Health Informatics and co-author of the study, told the Jersey Evening Post

“Our results build on evidence that indicates that polygenic risk scores do not have a role in public health screening programs,” Wald added.

“This research study rightly highlights that for many health conditions genetic risk scores alone may have limited usefulness, because other factors such as deprivation, lifestyles, and environment are also important,” clinical epidemiologist Raghib Ali, MD, CEO, Chief Investigator and Chief Medical Officer, Our Future Health UK, told The Guardian

Our Future Health is a collaboration between public, non-profit, and private sectors to create the UK’s largest health research program. The researchers in this endeavor intend to recruit over five million volunteers and use polygenic risk scores to develop innovative ways to prevent, detect, and treat disease. This program is funded by the UK’s National Health System (NHS).

“[Our] research program will be developing integrated risk scores that will take in all the important risk factors,” Ali explained. “We hope these integrated risk scores can identify people more likely to develop diseases, but this is a relatively new area of science and there are still unanswered questions around it.”

Danger of Misunderstanding DTC Genetic Tests

Here in the US, there have been news stories in recent years about the unreliability of certain genetic tests. Dark Daily covered these stories in previous ebriefs. News stories about the unreliability of genetic tests, particularly those marketed directly to consumers, reveal the problems that existing regulatory schemes have yet to address.

In “Consumer Reports Identifies ‘Potential Pitfalls’ of Direct-to-Consumer Genetic Tests,” we covered CR’s findings that though clinical laboratory and pathology professionals understand the difference between a doctor-ordered genetic health risk (GHR) test and a direct-to-consumer (DTC) genetic test, the typical genetic test customer may not. And that, misunderstanding the results of a DTC at-home genetic test can lead to confusion, loss of privacy, and potential harm.

Scientific American also covered the dangers of DTC testing in “The Problem with Direct-to-Consumer Genetic Tests,” in which the author notes that “despite caveats in ads and on packages, users can fail to understand their limitations,” and that “consumer-grade products are easily misconstrued as appropriate medical tests and create false reassurances in patients who could be at legitimate risk.”

Most clinical laboratory managers and pathologists are probably not surprised that the research performed at UCL shows that there are still issues surrounding genetic tests, particularly those marketed directly to consumers. While direct-to-consumer DNA tests can have some benefits, at this time, they are not always the best option for individuals seeking information about their personal risk for hereditary diseases.

—JP Schlingman

Related Information:

Over the Counter Genetic Tests in UK ‘Fail to Identify 89%’ of Those at Serious Risk

Performance of Polygenic Risk Scores in Screening, Prediction, and Risk Stratification: Secondary Analysis of Data in the Polygenic Score Catalog

3 Things to Know about At-home DNA Testing Kits

What Can At-home Genetic Tests Tell Me about My Health?

What are the Benefits and Risks of Direct-to-consumer Genetic Testing?

Genetic Risk Scores ‘Do Not Have a Place in Public Health Screening’

Consumer Reports Identifies ‘Potential Pitfalls’ of Direct-to-Consumer Genetic Tests

The Problem with Direct-to-Consumer Genetic Tests

Artificial Intelligence in the Operating Room: Dutch Scientists Develop AI Application That Informs Surgical Decision Making during Cancer Surgery

Speedy DNA sequencing and on-the-spot digital imaging may change the future of anatomic pathology procedures during surgery

Researchers at the Center for Molecular Medicine (CMM) at UMC Utrecht, a leading international university medical center in the Netherlands, have paired artificial intelligence (AI) and machine learning with DNA sequencing to develop a diagnostic tool cancer surgeons can use during surgeries to determine in minutes—while the patient is still on the operating table—whether they have fully removed all the cancerous tissue.

The method, “involves a computer scanning segments of a tumor’s DNA and alighting on certain chemical modifications that can yield a detailed diagnosis of the type and even subtype of the brain tumor,” according to The New York Times, which added, “That diagnosis, generated during the early stages of an hours-long surgery, can help surgeons decide how aggressively to operate, … In the future, the method may also help steer doctors toward treatments tailored for a specific subtype of tumor.”

This technology has the potential to reduce the need for frozen sections, should additional development and studies confirm that it accurately and reliably shows surgeons that all cancerous cells were fully removed. Many anatomic pathologists would welcome such a development because of the time pressure and stress associated with this procedure. Pathologists know that the patient is still in surgery and the surgeons are waiting for the results of the frozen section. Most pathologists would consider fewer frozen sections—with better patient outcomes—to be an improvement in patient care.

The UMC Utrecht scientist published their findings in the journal Nature titled, “Ultra-Fast Deep-Learned CNS Tumor Classification during Surgery.”

 “It’s imperative that the tumor subtype is known at the time of surgery,” Jeroen de Ridder, PhD (above), associate professor in the Center for Molecular Medicine at UMC Utrecht and one of the study leaders, told The New York Times. “What we have now uniquely enabled is to allow this very fine-grained, robust, detailed diagnosis to be performed already during the surgery. It can figure out itself what it’s looking at and make a robust classification,” he added. How this discovery affects the role of anatomic pathologists and pathology laboratories during cancer surgeries remains to be seen. (Photo copyright: UMC Utrecht.)

Rapid DNA Sequencing Impacts Brain Tumor Surgeries

The UMC Utrecht scientists employed Oxford Nanopore’s “real-time DNA sequencing technology to address the challenges posed by central nervous system (CNS) tumors, one of the most lethal type of tumor, especially among children,” according to an Oxford Nanopore news release.

The researchers called their new machine learning AI application the “Sturgeon.”

According to The New York Times, “The new method uses a faster genetic sequencing technique and applies it only to a small slice of the cellular genome, allowing it to return results before a surgeon has started operating on the edges of a tumor.”

Jeroen de Ridder, PhD, an associate professor in the Center for Molecular Medicine at UMC Utrecht, told The New York Times that Sturgeon is “powerful enough to deliver a diagnosis with sparse genetic data, akin to someone recognizing an image based on only 1% of its pixels, and from an unknown portion of the image.” Ridder is also a principal investigator at the Oncode Institute, an independent research center in the Netherlands.

The researchers tested Sturgeon during 25 live brain surgeries and compared the results to an anatomic pathologist’s standard method of microscope tissue examination. “The new approach delivered 18 correct diagnoses and failed to reach the needed confidence threshold in the other seven cases. It turned around its diagnoses in less than 90 minutes, the study reported—short enough for it to inform decisions during an operation,” The New York Times reported.

But there were issues. Where the minute samples contain healthy brain tissue, identifying an adequate number of tumor markers could become problematic. Under those conditions, surgeons can ask an anatomic pathologist to “flag the [tissue samples] with the most tumor for sequencing, said PhD candidate Marc Pagès-Gallego, a bioinformatician at UMC Utrecht and a co-author of the study,” The New York Times noted. 

“Implementation itself is less straightforward than often suggested,” Sebastian Brandner, MD, a professor of neuropathology at University College London, told The Times. “Sequencing and classifying tumor cells often still required significant expertise in bioinformatics as well as workers who are able to run, troubleshoot, and repair the technology,” he added. 

“Brain tumors are also the most well-suited to being classified by the chemical modifications that the new method analyzes; not all cancers can be diagnosed that way,” The Times pointed out.

Thus, the research continues. The new method is being applied to other surgical samples as well. The study authors said other facilities are utilizing the method on their own surgical tissue samples, “suggesting that it can work in other people’s hands.” But more work is needed, The Times reported.

UMC Utrecht Researchers Receive Hanarth Grant

To expand their research into the Sturgeon’s capabilities, the UMC Utrecht research team recently received funds from the Hanarth Fonds, which was founded in 2018 to “promote and enhance the use of artificial intelligence and machine learning to improve the diagnosis, treatment, and outcome of patients with cancer,” according to the organization’s website.

The researchers will investigate ways the Sturgeon AI algorithm can be used to identify tumors of the central nervous system during surgery, a UMC Utrecht news release states. These type of tumors, according to the researchers, are difficult to examine without surgery.

“This poses a challenge for neurosurgeons. They have to operate on a tumor without knowing what type of tumor it is. As a result, there is a chance that the patient will need another operation,” said de Ridder in the news release.

The Sturgeon application solves this problem. It identifies the “exact type of tumor during surgery. This allows the appropriate surgical strategy to be applied immediately,” the news release notes.

The Hanarth funds will enable Jeroen and his team to develop a variant of the Sturgeon that uses “cerebrospinal fluid instead of (part of) the tumor. This will allow the type of tumor to be determined already before surgery. The main challenge is that cerebrospinal fluid contains a mixture of tumor and normal DNA. AI models will be trained to take this into account.”

The UMC Utrecht scientists’ breakthrough is another example of how organizations and research groups are working to shorten time to answer, compared to standard anatomic pathology methods. They are combining developing technologies in ways that achieve these goals.

—Kristin Althea O’Connor

Related Information:

Ultra-fast Deep-Learned CNS Tumor Classification during Surgery

New AI Tool Diagnoses Brain Tumors on the Operating Table

Pediatric Brain Tumor Types Revealed Mid-Surgery with Nanopore Sequencing and AI

AI Speeds Up Identification Brain Tumor Type

Four New Cancer Research Projects at UMC Utrecht Receive Hanarth Grants

Rapid Nanopore Sequencing, Machine Learning Enable Tumor Classification during Surgery

;