Artificial intelligence is increasingly being used in healthcare, with patients turning to AI chatbots like ChatGPT for information and advice. A recent study by the Sylvester Comprehensive Cancer Center examined how well ChatGPT 3.5, a widely available version of OpenAI’s chatbot as of July 2024, responds to questions about blood cancer.
The research involved four hematology-oncology physicians who evaluated ChatGPT’s answers to ten questions that reflected typical patient concerns. The questions ranged from general topics such as side effects of chemotherapy to more specific queries about newer treatments like BCL-2 inhibitors.
Results showed that ChatGPT 3.5 performed better on general cancer-related questions than on those involving new therapies or specific treatment approaches. The chatbot received an average score of 3.38 out of 5 for general questions but only 3.06 for newer therapies. None of its responses received a perfect score from any evaluator.
Justin Taylor, M.D., senior author of the study and physician at Sylvester, cautioned patients: “I would warn patients to have some skepticism, especially about answers dealing with specific types of cancer and treatments, and check with their doctor.”
A key limitation identified was that ChatGPT 3.5’s knowledge base only included data up until 2021, meaning it could not provide information on medical advances made after that year. Dr. Taylor explained the importance of human expertise: “When new drugs or research findings emerge, oncologists check in with their colleagues, discuss the implications and think about how to adapt them to their patients.” He added, “Chatbots can’t provide that kind of nuance and personalized understanding.”
Despite these limitations, Dr. Taylor noted that AI tools can still be useful for organizing thoughts before appointments or identifying primary sources for further discussion with healthcare professionals. However, he emphasized: “Physician oversight remains essential for vetting AI-generated medical information before patient use.”
Patients are advised to consult their doctors regarding any information obtained from AI or online sources in order to receive guidance tailored to their individual health needs.
