In a new study published in Communications Biology, researchers have called attention to variable results produced by different direct-to-consumer gut microbiome testing kits, cautioning consumers and clinicians when interpreting and acting on the test results.
Direct-to-consumer gut microbiome testing kits provide information on an individual’s microbiome composition, and many use these results to generate health assessments. However, the accuracy of these tests remains uncertain and no study has systematically evaluated their performance until now.
In their study, the researchers sent the same anonymized homogenized bulk stool sample to multiple companies to compare the results and reproducibility between companies and workflows. They also sent the sample to the same company multiple times to evaluate internal reproducibility.
“The sample came back with different results from these companies and even within the same company,” said Diane Hoffman, Jacob A. France Professor of Health Law at the University of Maryland School of Law and a co-author of the study, during a press briefing last week.
“The hype doesn’t really match the evidence,” she said. “These tests often have limited evidence behind them, especially when it comes to informing clinical decisions or even basic dietary recommendations. The marketing can be questionable and consumers can end up misinterpreting or over-trusting results that are not very reliable.”
The researchers suggest that a lack of standardized sampling procedures, sample processing, and different analysis workflows used by testing companies may contribute to these discrepancies.
“Different companies are using different workflows, so the results are going to differ,” said Scott Jackson, a co-author of the study, previous leader of the Complex Microbial Systems Group at the National Institute of Standards and Technology (NIST), and founder of NEST Consulting. “There is bias in every step along the way. And these can impact the results.”
“Even though [the test kits] fall under CLIA for regulatory purposes … analytical validation is really tough in this space because there is no standardization,” said Hoffman.
The authors suggest that standardizing sampling protocols and testing methods could help improve the accuracy of direct-to-consumer microbiome testing.
Sources of variability
Sampling
Variability can be introduced via the sampling method and tool, such that your starting sample can be very different, explained Stephanie Servetas, acting group leader of the Complex Microbial Systems Groups at NIST. “Some kits ask you to swab from used toilet paper, some send collectors, and some send buffers and some do not,” she said. Samples are then sent back to the company at room temperature, sometimes this involves three to four days of travel, and then the kit might sit at the company for a further time. It’s also unclear how samples are stored. “This all affects the starting sample and thus the results,” she said.
DNA extraction
“The DNA Extraction and data analysis is a huge source of bias,” said Jackson. “None of them are 100 percent efficient at extracting DNA across all cell types. Different DNA extraction kits will extract different populations of DNA from different types of bacteria depending on how the extraction kits lyse the bacteria. There’s a paper coming out that looked at this systematically and shows a systemic correlation between that ratio and the method used to lyse the cells. The ratio is not diagnostic.”
Bioinformatics analysis
“The second largest source of variability is the analysis,” continued Jackson. “Once the raw data comes off the sequencer, it has to undergo bioinformatics workflows that involve lead mapping to reference genomes, and there are over 100 software tools that have been developed by academics and industry to interpret microbiome data. They all use different algorithms and databases and different statistics and assumptions, and they will all produce a very different result.”
“These companies largely hang their hat on their proprietary analysis workflow. So, whatever data analysis they’re doing, it is proprietary to the company, but by that very nature, they are all doing it very differently.”
Different companies are also using different cut-offs for what they’re including in their final report, said Servetas, whereas others provided a very extensive list of what’s present.
Recommendations for consumers
“You can use these tests to track lifestyle changes that affect microbiome, but they cannot provide meaningful or actionable clinical diagnoses or information,” said Servetas.
“Physician often don’t understand the results and how they are connected to overall health,” said Hoffman, and in some cases, this can seed distrust in clinicians versus the trust people are putting into these companies.
For consumers, the researchers recommend reviewing a company’s sampling and analysis workflow, as well as reviewing their publications when deciding on a testing kit.
Gut microbiome testing is still a new field
“Next-generation sequencing technology has advanced dramatically in terms of price reduction and data output, but that hasn’t translated necessarily to better microbiome measurements,” said Jackson. “This is still a relatively new field…we’re still learning how to do these measurements better, and it’s still early as a scientific field—we’re just not there yet.”
“There needs to be rigorous clinical studies that are designed properly to be able to home in on biomarkers that are indicators of disease, and that’s tricky to do when measuring thousands of things in one sample,” he said.
The researchers plan to conduct follow-up studies where they will review the raw sequencing data from the different companies and reanalyze it to determine whether they can get the results to line up.
