Humans Beat AI in Making Personalized Medicine Decisions

If the body can no longer repair certain genetic mutations itself, cells begin to grow unchecked, producing a tumor. The crucial factor in this phenomenon is an imbalance of growth-inducing and growth-inhibiting factors, which can result from changes in oncogenes. Precision oncology, a specialized field of personalized medicine, leverages this knowledge by using specific treatments, such as low-molecular weight inhibitors and antibodies, to target and disable hyperactive oncogenes.

The first step in identifying which genetic mutations are potential targets for treatment is analyzing the genetic makeup of the tumor tissue: Determining the molecular variants of the tumor DNA that are necessary for precision diagnosis and treatment. Clinicians then use this information to craft individual treatment recommendations. In especially complex cases, this requires knowledge from various fields of medicine. Experts from the fields of pathology, molecular pathology, oncology, human genetics, and bioinformatics work together to analyze which treatments seem most promising based on the latest studies. 

Can artificial intelligence help with treatment decisions?

Damian Rieke, MD, a physician at Charité – Universitätsmedizin Berlin, Ulf Leser, Dr rer nat, professor and Xing David Wang, a PhD student, both of Humboldt-Universität zu Berlin, and Manuela Benary, Dr rer nat, a bioinformatics specialist at Charité, wondered whether artificial intelligence might be able to help at this juncture. In a study recently published in the JAMA Network Open, they teamed up with other researchers to examine the possibilities and limitations of large language models such as ChatGPT in automatically scanning scientific literature to identify and select personalized treatments.

“We prompted the models to identify personalized treatment options for fictitious cancer patients and then compared the results with the recommendations made by experts,” Rieke explains. His conclusion: “AI models were able to identify personalized treatment options in principle, but they weren’t even close to the abilities of human experts.”

The team created ten molecular tumor profiles of fictitious patients for the experiment. A human physician specialist and four large language models were then tasked with identifying a personalized treatment option. These results were presented to the members of the Molecular Tumor Board (MTB) for assessment, without them knowing where which recommendation came from.

Improved AI models hold promise for future use

“There were some surprisingly good treatment options identified by AI in isolated cases,” Benary reports. “But large language models perform much worse than human experts.” Beyond that, data protection, privacy, and reproducibility may become challenging when using AI with real-world patients, Benary notes.

Still, Rieke is fundamentally optimistic about the potential uses of AI in medicine: “In the study, we also showed that the performance of AI models is continuing to improve as the models advance. This could mean that AI can provide more support for even complex diagnostic and treatment processes in the future, as long as humans are the ones to check the results generated by AI and have the final say about treatment.”

- This press release was originally published on the Charité – Universitätsmedizin Berlin website