Exploring chatbot applications in pancreatic disease treatment: potential and pitfalls

INTRODUCTION

Over recent decades, advancements in artificial intelligence (AI) have revolutionized various domains, including the development of conversational agents known as chatbots. These systems utilize AI and natural language processing (NLP) to interpret user queries and generate automated responses, mimicking human dialogue^[1]. Chatbots are primarily built on two components: a general-purpose AI framework and a user-friendly chat interface^[2]. By leveraging large language models (LLMs), chatbots demonstrate conversational reasoning, producing contextually appropriate responses^[3].

The public release of Chat Generative Pretrained Transformer (ChatGPT) in November 2022 marked a significant milestone, accelerating the proliferation of similar AI-driven tools across industries^[4]. However, most LLMs, including ChatGPT, are trained predominantly on non-medical datasets^[5].

This limitation has sparked widespread discussions regarding the use of such tools in sensitive fields such as medicine, raising ethical concerns about bias reinforcement, potential plagiarism, misinformation, and the reliability of AI-generated content^[6,7].

Despite these challenges, researchers have begun investigating the quality and reliability of chatbot-generated medical information^[8]. Preliminary studies have compared the accuracy and readability of responses from various chatbots across general medical topics and specific specialties^[9].

Recent efforts include evaluations of chatbots’ performance on topics such as cancer care, ophthalmology, and other specialized domains^[10-14].

Despite extensive research on AI-generated information in medical fields, there is a lack of data evaluating the performance of Large Language Models (LLMs) specifically addressing pancreatic cancer treatment. Pancreatic cancer is burdened by a poor prognosis related to the almost always advanced stage of the disease at diagnosis. The chances of cure are primarily linked to the timeliness of surgical intervention - which is, however, associated with a high morbidity rate - and to perioperative chemotherapy, whose protocols are continuously evolving due to advances in oncological research. This underscores the importance of strictly adhering to clinical guidelines to ensure that each patient receives the best possible access to treatment^[15]. This study aims to bridge that gap by evaluating the accuracy and reliability of chatbot-generated information on pancreatic cancer, comparing multiple LLMs under standardized clinical conditions. Specifically, we analyzed chatbot responses to queries posed by specialized clinicians on topics including risk factors, diagnosis, and management of resectable, borderline resectable, locally advanced, and metastatic pancreatic cancer. Our primary objectives were to assess the accuracy of AI-provided medical information, evaluate its concordance with international guidelines, and determine the comprehensiveness of the content delivered.

MATERIALS AND METHODS

Questionnaire selection & testing

A panel of pancreatic surgeons meticulously developed the questionnaire. The questionnaire consisted of 14 questions evenly distributed across seven topics: epidemiology (n = 2), diagnosis (n = 2), treatment options for radiologically resectable disease (n = 2), borderline resectable disease (n = 2), locally advanced disease (n = 2), metastatic pancreatic cancer (n = 2), and best supportive care (n = 2) [Figure 1]. The process began with a thorough review of the most widely used clinical guidelines for pancreatic cancer:

Figure 1. Questionnaire performed by chatbots.

(1) Clinical Practice Guidelines in Oncology (NCCN guidelines®) for Pancreatic Adenocarcinoma^[16].

(2) ESMO Clinical Practice Guideline for diagnosis, treatment, and follow-up for pancreatic cancer^[17].

(3) Pancreatic cancer in adults: diagnosis and management from the National Institute for Health and Care Excellence^[18].

(4) Clinical Practice Guidelines for Pancreatic Cancer 2022 from the Japan Pancreas Society: a synopsis^[19].

Each question was derived directly from the clinical guidelines, and phrased using conventional medical terminology appropriate for a surgeon. Before querying, all chatbots were given the same introductory prompt: “I am a pancreatic surgeon, and I’d like to know more about pancreatic cancer”. This was designed to stimulate the perspective of a specialist and trigger the chatbot to provide a precise recommendation rather than generic or broad information related to pancreatic cancer. Follow-up prompts were not used.

RESULTS

A total of 168 questions were posed to the chatbots. All chatbots answered every question.

Inter-rater agreement in the Likert-scale ratings was substantial, with a Cohen’s kappa of 0.659 (Z = 12.5, P < 0.001), and a simple percentage agreement of 76.8%, indicating a high degree of consistency between the two evaluators.

Table 1 summarizes the distribution of correct answers, with 71.5% of responses scored as either 3 or 4. The most accurate chatbot was Anthropic Claude 3 Haiku Version 3.5, which provided 38/120 (31.6%) correct answers, followed by ChatGPT-4 with 33/120 (27.5%). Of the 54 answers receiving the highest score of 4, Anthropic Claude 3 Haiku Version 3.5 provided the majority (22/54, 40.7%), followed closely by ChatGPT-4 (21/54, 38.9%).

Table 2 highlights the best of the three answers to each question. ChatGPT-4 achieved the highest number of correct and comprehensive responses (11/14, 78.6%), followed by Anthropic Claude 3 Haiku Version 3.5 (10/14, 71.4%) and Perplexity AI (6/14, 42.9%). However, ChatGPT-4 exhibited the greatest variability in responses, with 10/14 (71.4%) answers rated inconsistently by evaluators. Surgical treatment options and systemic therapies received the lowest average scores, with most responses scoring below 3. ChatGPT-4 produced the longest responses on average, while Anthropic Claude 3 Haiku Version 3.5 offered the highest proportion of precise recommendations.

ChatGPT-4 provided the highest number of guideline-cited answers (12/42, 28.6%), followed by Anthropic Claude 3 Haiku Version 3.5 (10/42, 23.8%) and Personal Intelligence by Inflection AI (7/42, 16.7%). Overall, only 18.5% of responses referenced guidelines, indicating room for improvement in evidence-based citations.

All LLMs delivered responses exceeding the Flesch-Kincaid reading level of 6 recommended by the National Institute of Health (NIH) and the average American reading level of 8^[22]. The reading levels of the LLMs’ answers ranged from 9,4 (Claude, most accessible) to 20 (PiAI, least accessible). On average, all four LLMs demonstrated an average reading score exceeding 12, indicative of advanced (college-level) reading proficiency. The most accessible LLM was Perplexity (Flesch-Kinkaid grade level 12,3), while the least accessible was PIAI (15,6, P < 0,007) [Table 3].

DISCUSSION

The rapid adoption of chatbots worldwide has positioned them as potentially transformative tools across various domains, including medicine. Their user-friendly interfaces and ability to synthesize vast amounts of knowledge have generated significant interest in their application for research and clinical decision making. However, as their role in healthcare expands, concerns about their accuracy, reliability, and safety become increasingly pertinent. This study aimed to evaluate the safety and reliability of chatbot-generated responses in addressing pancreatic cancer, a highly specialized and critical medical field.

Consistent with prior research, our findings indicate that although chatbots frequently provide accurate and reliable responses, a significant proportion of their outputs remain incomplete or inaccurate. The apparent discrepancy between the overall correctness rate (71.5%) and the distribution of errors across individual LLMs reflects the substantial variability in performance among different models. While some LLMs demonstrated relatively high accuracy, others generated a markedly higher number of incorrect or incomplete responses. These limitations render chatbots unsuitable for independent medical consultation without expert oversight. For example, our results are consistent with those of Hermann et al., who demonstrated ChatGPT's ability to accurately discuss cervical cancer prevention but highlighted shortcomings in areas requiring nuanced understanding, such as diagnosis and treatment^[23]. Similar trends have been observed in other medical specialties, including head and neck surgery, bariatric surgery, and oncology, further underscoring the variability in chatbot performance across different domains^[24-26].

A notable observation in this study was the variability in chatbot responses, even when identical queries were posed. This inconsistency was particularly evident in ChatGPT, likely due to differences in the specific versions used. Although all researchers employed the same underlying model (LLM Model 4), disparities were observed between the paid version, the free 4.0 version, and the free 4.0 Mini version. The paid version consistently outperformed its free counterparts, providing superior responses in 10 out of 14 cases. However, overlapping response quality occurred in 8 of these cases, demonstrating that free versions occasionally match the performance of paid versions in certain contexts. The decision to include both the free and paid versions of ChatGPT was made to reflect real-world usage scenarios, where clinicians may access different platforms with varying levels of availability and performance. For this reason, the analysis was not restricted to the most advanced version of a single model, nor was it standardized based on subscription level. The readability scores highlighted that LLMs use advanced language, well above the comprehension level of the general population. In this context, the responses may be easily understood by medical personnel, such as the pancreatic surgeons asking the questions, but could be challenging for patients seeking information about pancreatic cancer. The complexity of the language may, however, be influenced by the initial context, as the chatbot was consistently informed at the start of the questionnaire that the user was a pancreatic surgeon. Therefore, further evaluations are needed to assess the feasibility of using chatbots as an information source for the general population.

Another key finding was the inconsistency in the sources utilized by chatbots. Despite having access to publicly available guidelines, such data was incorporated in only 18.5% of responses. Another critical finding was the inconsistency in the sources utilized by chatbots. Despite having access to publicly available guidelines, such sources were explicitly referenced in only 18.5% of responses. Moreover, chatbots occasionally incorporate information from less authoritative platforms, such as patient blogs or general online content, which lack the rigor of peer-reviewed scientific literature. This limitation underscores a key challenge for current-generation LLMs: their inability to transparently assess or prioritize the reliability of their training data. Such shortcomings raise concerns about their suitability for specialized medical applications, where adherence to evidence-based guidelines and credible sources is paramount^[27].

Previous studies have documented inaccuracies and fabrications in chatbot-generated responses, a phenomenon referred to as “hallucination”^[28]. This issue arises when chatbots generate plausible-sounding but factually incorrect responses, making errors less apparent to users. Our study found this issue to be more pronounced in responses to highly specialized queries, where chatbots deviated from established guidelines and relied on less authoritative sources. These compromises in source selection and accuracy can pose significant risks, particularly in high-stakes fields like oncology care. In clinical settings, such errors could lead to misinformation with serious consequences. Therefore, we suggest future work should focus on domain-specific fine-tuning and integration with up-to-date clinical databases to mitigate hallucination risk.

Our findings emphasize the need for caution in integrating chatbots into clinical practice. While the potential of these tools to revolutionize healthcare is undeniable, substantial advancements are required to ensure their safety and reliability. At present, chatbot responses are useful for providing a general and superficial overview of a scientific topic, but they are not yet sufficiently trained to deliver in-depth answers to precise or highly specific questions. Training LLMs with domain-specific datasets curated by medical experts could significantly enhance their performance and address existing limitations. Although current LLMs are not yet ready for autonomous clinical decision making, they could be integrated as decision-support tools. With expert oversight, these systems might assist in synthesizing clinical data, guiding diagnostic pathways, and recommending treatment strategies according to the most updated guidelines and resources. Such improvements would enable chatbots to serve as valuable tools for disseminating medical knowledge, supporting healthcare professionals, and enhancing patient education^[20]. Soon, LLMs might play a crucial role in enhancing, rather than replacing, the clinician’s expertise^[29]. In any case, especially in light of the results obtained in the present study, it is mandatory to use chatbots with a critical perspective - avoiding the assumption that their answers are definitive truths, and instead placing them within a broader context of research, guidelines evaluation, and the latest scientific evidence.

This study is not without limitations. First, inter-reviewer variability may have influenced the evaluation of chatbot responses, as subjective judgment was inherent to the assessment process. Second, in prompting, different user identities (e.g., general physicians or patients) might elicit different responses that would benefit from further research. Furthermore, we employed standardized and concise questions, without resorting to advanced prompt engineering techniques. It is plausible that the use of more elaborate prompts could have improved the quality and completeness of the responses. We also did not explore multi-turn interactions or follow-up questioning, which may positively influence the accuracy and clinical usefulness of chatbot outputs. We believe that, to ensure the safe use of chatbots in the medical field, the initial response should already be accurate and based on reliable sources, without the need for repeated prompting to obtain an acceptable answer. Third, given the qualitative variability of chatbot outputs, we opted for a descriptive approach to provide a preliminary overview of performance differences. Fourth, the main use of free chatbot versions, which are known to have reduced data processing capabilities compared to paid versions, may have impacted our findings. Future research should include a broader range of chatbot models and focus on developing objective evaluation criteria to provide more robust and reproducible insights.

While the rapid evolution of artificial intelligence presents exciting possibilities for healthcare, our findings underscore the need for continued scrutiny and refinement of chatbot technologies. Ensuring their safety, reliability, and adherence to evidence-based practices is essential before their widespread adoption in clinical settings.

In conclusion, chatbots are emerging as increasingly efficient tools for answering queries rapidly, offering a convenient alternative to the time-consuming process of searching through traditional literature. However, the extensive and often non-curated datasets used by LLMs pose significant risks, as unchecked outputs can lead to misinformation and errors.

The rapid advancements in LLM technology are promising, but their successful integration into clinical practice will require significant improvements in reliability, transparency, and adherence to evidence-based guidelines. If these challenges are addressed, LLMs could transform from supplementary tools into essential assets in the surgical field.