Evaluating the efficacy of major language models in providing guidance for hand trauma nerve laceration patients: a case study on Google's AI BARD, Bing AI, and ChatGPT

INTRODUCTION

Exponential advancements in AI have catalyzed the emergence of LLMs, which are gaining interest for their abilities to synthesize large swaths of information and produce comprehensible answers to most queries^[1,2].

ChatGPT has proven to answer complex medical queries, which could revolutionize healthcare for patients with limited access to medical professionals^[1]. Hand trauma, specifically nerve lacerations, is common and can cause severe impairment if mismanaged^[3]. Timely and accurate information is imperative but can be limited, particularly in rural and low-resource settings^[4]. Opportunistically, AI-powered language models could fill this gap as potential medical guidance providers.

ChatGPT has dominated the mainstay of articles determining the suitability of LLMs in medical practice. With the launch of other AI tools, specifically Google’s AI BARD and BingAI, their comparative performances should be evaluated. Therefore, this study evaluated Google’s AI BARD, BingAI, and ChatGPT for providing accurate and relevant information to patients presenting with hand trauma nerve lacerations. Using objective and subjective metrics, we assessed contextual understanding and recommendation suitability. This comparison will highlight strengths and weaknesses, informing improvements in AI-driven guidance for hand trauma care.

CASE REPORT

The authors evaluated the suitability of three LLMs-Google’s AI BARD, BingAI, and ChatGPT-4 by their capacity to interpret medical literature, extract relevant data, and produce precise, intelligible, and contextually suitable clinical advice. Their responses were also compared against established clinical guidelines. Additionally, the analysis will encompass the efficiency, dependability, potential biases, and ethical implications affiliated with each LLM within the realm of nerve injury management.

A set of simulated patient-perspective queries concerning digital nerve injury diagnosis and management were presented to ChatGPT, BARD, and Bing AI. Responses generated were compared to existing clinical guidelines and literature. Additionally, a panel of plastic surgery residents and Board-certified Plastic Surgeons with extensive peripheral nerve injury expertise evaluated the responses with a Likert scale. Assessment criteria included accuracy, comprehensiveness, and provision of relevant information sources.

To maintain consistency and precision, the same author (BL) documented the initial response each LLM provided for every question, avoiding further clarifications or alterations. Questions were crafted to avoid grammatical errors or ambiguity and were simultaneously inputted using separate accounts for OpenAI, Google, and Microsoft, granting access to ChatGPT-4, BARD, and Bing AI, respectively.

We compiled a dataset of hand trauma nerve laceration cases, encompassing diverse scenarios, symptoms, and treatment alternatives. Each language model's efficacy was assessed on a Likert scale [Table 1] based on its capacity to offer patient guidance accurately and effectively. Evaluation criteria comprised the subsequent elements:

Accuracy: the correctness and dependability of the information supplied by the language models. Comprehensibility: the patients' ability to readily understand the provided information and directions. Empathy and tone: the language models' capacity to convey empathy and sustain a suitable tone. Timeliness: the language models' speed in generating pertinent information.

Additionally, the readability and reliability of the LLMs’ responses were evaluated using specific metrics. The Flesh Reading Ease Score, Flesch-Kincaid Grade Level, and the Coleman-Liau Index were employed for readability assessment, whereas the DISCERN score was utilized for reliability. The results are consolidated in Table 2 and subsequently subjected to a t-test [Table 3] for statistical significance appraisal.

Figure 1 introduced the first scenario to the LLMs, reading “Hi Large Language Model, I am a 23-year-old male who has cut his right index finger with a knife. I am right hand dominant and a professional guitar player. I do not have any sensation on my right index fingertips. What do you think has occurred and what treatment do I require?” ChatGPT's response, which highlighted the prompt in red and warned of content policy violations, immediately suggested consulting professionals for clinical advice^[5]. Nonetheless, it identified the correct affected nerve and outlined damage mitigation steps^[6]. Notably, ChatGPT considered the user's right-handedness and occupation, urging prompt expert help and personalized care. It concluded by discouraging risky activities and reiterating the need for professional assistance. Google’s BARD presented a response between ChatGPT and Bing AI in quality, suggesting nerve injury and advising immediate physician consultation. It also recommended basic first-aid and concluded with insights into the injury’s nature, prognosis, and potential therapies^[7]. Bing AI proffered a comparable response, elucidating its primary diagnosis of nerve damage, advocating for professional consultation and delineating possible treatment methods. Unlike ChatGPT, Bing AI did not propose an intermediate care model.

Figure 1. Hi Large Language Model, I am a 23-year-old male who has cut his right index finger with a knife. I am right hand dominant and a professional guitar player. I do not have any sensation on my right index fingertips. What do you think has occurred and what treatment do I require?

Figure 2 aimed to evaluate the models’ ability to follow up on previous queries (recall ability) with additional information. This read “Same patient as the previous question, what surgical procedure do you recommend? Do I require any diagnostic test prior to surgical intervention?” ChatGPT reemphasized its ill-suitedness to provide recommendations and encouraged users to consult a physician. Nonetheless, it enumerated diagnostic tests typically conducted for injury diagnosis and severity assessment, concluding its response by mentioning common treatment modalities^[8,9]. BARD furnished a concise response, advocating for surgical repair and what it entailed, and listed identical diagnostic evaluations as ChatGPT did. It expounded on the repair prognosis being dependent on various factors, with earlier intervention often producing better outcomes. Augmenting ChatGPT’s reply, BARD outlined postoperative rehabilitation strategies, encompassing orthotic support and physical therapy for functional recovery^[10]. Unlike ChatGPT, BARD did not stress the importance of seeking expert counsel. Bing AI acknowledged its limitations by suggesting consultation with a professional and highlighting the importance of diagnostic assessments. However, it failed to delineate precise examinations and suitable therapeutic alternatives. Ultimately, it offered an indistinct summary compared to ChatGPT and BARD.

Figure 2. Same patient as the previous question, what surgical procedure do you recommend? Do I require any diagnostic test prior to surgical intervention?

In Figure 3, the inquiry “If I have completely lacerated my digital nerve, what is the likely outcome of being completely repaired? Provide 5 high-level evidence studies to support your answer.” sought to assess the models’ capacity to supply pertinent references and predict surgical outcomes. ChatGPT delivered a vague response, stating that outcomes are generally favorable and contingent on factors such as injury severity, early surgical intervention and repair methodology^[11]. The bulk of its reply addressed the query regarding high-level references, failing to provide five legitimate citations, and recommended users search reputable databases. It concluded by recommending consulting a healthcare professional. BARD merely asserted its inability to assist due to it being a language model, while Bing AI offered two systematic reviews and another article on recovery rates from nerve injuries, also advising for a healthcare consultation. The first two references, despite existing in the literature, share a primary author and cover similar subjects^[12,13], while the third reference displayed the paper’s title, omitting the authors and publication date. Furthermore, it failed to supply five references.

Figure 3. If I have completely lacerated my digital nerve, what is the likely outcome of being completely repaired? Provide 5 high-level evidence studies to support your answer.

As shown in Figure 4, the prompt “Following digital nerve repair surgery, what is the timeframe for the complete return of sensation to the finger?” aimed to analyze the models’ abilities to predict prognosis following a surgical procedure. ChatGPT commenced by noting the variable timeline is contingent on multiple factors. It delineated the average nerve regeneration rate, emphasizing the gradual process of regaining sensation, while also warning potential incomplete recovery^[14,15]. ChatGPT finally advised adherence to healthcare professional guidance, encompassing hand therapy, sensory re-education, and pain management^[16]. BARD generated a response akin to ChatGPT's, offering an approximate recovery timeframe of six months to two years, depending on injury severity and patient health, while similarly cautioning against incomplete sensation restoration^[17,18]. It expounded on three factors influencing recovery duration and emphasized adhering to healthcare professional directives^[19]. Bing AI proposed an alternative recovery timeframe of three to six months while cautioning that not all individuals achieve full recovery^[3]. Interestingly, it cited a study to substantiate its assertion and furnished two additional links for users seeking further research.

Figure 4. Following digital nerve repair surgery, what is the timeframe for the complete return of sensation to the finger?

The final input “Same patient, what are my options if I don’t have surgery?” as shown in Figure 5, assessed the models’ ability to offer a breadth of therapeutic options. Consistent with prior responses, ChatGPT advised seeking a healthcare professional's guidance. Initially, it elucidated potential complications arising from non-surgical approaches followed by enumerating alternative treatments^[20]; however, it ultimately advocated for surgical management, attributing this to enhanced patient outcomes. BARD devised several recommendations for non-clinical settings, subsequently accentuating the potential inadequacy of these interventions and suggesting surgical measures should symptoms persist. The response culminated by listing the ramifications of forgoing surgery and urging users to consult their doctor for optimal management^[20]. Bing AI presented and briefly explained distinct alternative therapies, underscoring the variability of patient outcomes and that not all nerve lacerations necessitate surgical intervention^[21]. Bing AI also cited a systematic review to support its assertion and furnished at least four additional links for users to further study this topic. Unfortunately, the hyperlink to the systematic review was erroneous, raising concerns about Bing AI’s credibility. It also failed to list possible complications of adopting non-surgical management, resulting in a less comprehensive response.

Figure 5. Same patient, what are my options if I don’t have surgery?

In assessing readability, ChatGPT and BARD performed similarly [Table 2]. ChatGPT achieved the highest Flesch-Kincaid Grades and Coleman-Liau index scores, while BARD scored the highest Flesch Reading Ease Score. For reliability, ChatGPT marginally outperformed Bing AI, with BARD scoring the lowest. The t-test indicated that only six of the twelve comparisons were statistically significant. All the DISCERN score comparisons were statistically insignificant, meaning no LLM was necessarily more reliable than the other. Additionally, the readability for Bing AI vs. ChatGPT lacked statistical significance, suggesting that they exhibited similar comprehensibility.

DISCUSSION

Overall, ChatGPT provided the most comprehensive and comprehensible information, generating an extensive array of management solutions while incorporating contextual data, all in a reader-friendly manner. BARD’s answers were similar in structure to ChatGPT’s, albeit lacking the same level of detail and clarity. Bing AI struggled to attain similar levels of ChatGPT and BARD’s comprehensibility and easiness of understanding, often utilizing technical language with frequent statistical references and lacking succinct summaries which came across as unempathetic and impersonal, as reflected in its Flesch Reading Ease Score. However, its consistent provision of sources compensates for its shortcomings, resulting in its comparable DISCERN score to ChatGPT’s.

ChatGPT and BARD demonstrated superior comprehensibility and user-centricity to Bing’s AI, making them more suitable for improving public comprehension of nerve injury management. Overall, the accuracy of the three LLMs was insufficient for use as automated diagnostic tools supporting healthcare professionals, as they neglected to reference key peripheral nerve injury guidelines or high-quality research^[22]. Additionally, the LLMs omitted experimental treatments like stem cell therapy and photochemical tissue bonding, indicating a limitation in their algorithms' capacity for generating innovative solutions^[23]. Despite this, as a clinical practice tool, they could ensure that patients are not misled or provided an unfeasible option.

In terms of information depth, ChatGPT provided supplementary data and enhanced its primary response when asked, outperforming BARD. Although BARD’s comprehensiveness was comparable to ChatGPT, it failed to respond to the third prompt, severely impacting its readability and reliability scores. Moreover, ChatGPT presented the most thorough rationale for each suggestion, bolstering its credibility. BARD closely followed, but its explanations lacked ChatGPT's depth. Bing AI delivered the least detailed responses, which were sometimes not the gold standard. For example, it suggested exercise as an alternative treatment but neglected to specify that aerobic exercise is the most optimal form for addressing nerve injuries^[24]. Nevertheless, Bing AI offered the most diverse range of treatment alternatives. Therefore, while Bing AI lacked depth, ChatGPT and BARD were limited in breadth. Notably, all three LLMs concentrated on layperson first aid, offering limited information for healthcare professionals and academics. They omitted management algorithms for nerve injuries, for example, when Bhandari (2019) recommends immediate surgery for penetrating trauma with neurological symptoms but conservative management for blunt trauma. The LLMs also neglect to address Seddon and Sunderland's categorizations of peripheral nerve injuries, which impact management^[8,25,26]. This oversight may be attributed to the phrasing of the queries, as the LLMs could be presuming the authors are non-medical professionals, consequently yielding less scholarly and comprehensive replies. Considering this is the first study to compare these LLMs and on this topic, further research should seek to rectify these deficiencies.

Literature consistency was flagged by many studies investigating ChatGPT in the past, so the comparative performance in generating references was pertinent to this study. Bing AI demonstrated superior consistency compared to BARD and ChatGPT, often supplying relevant hyperlinks to fact-check its claims. Meanwhile, BARD failed to produce high-level references and ChatGPT only recommended databases for users to search or generated aberrant references. Despite this, Bing AI primarily cited health websites over scholarly articles and directed users to irrelevant web pages, resulting in a higher DISCERN score than BARD’s but failed to surpass ChatGPT’s. Despite these constraints, LLMs’ rapid information retrieval and summarization capacity make them attractive for patients who are gathering information about emerging medical issues.

Regarding user-friendliness and comprehensibility, ChatGPT and BARD exhibited comparable success. ChatGPT’s responses were succinct and devoid of technical jargon, rendering them advantageous for individuals with limited expertise, such as junior medical staff or patients. Additionally, ChatGPT acknowledged the user’s occupation and hand dominance, personalizing the interaction. BARD displayed a similar syntactical approach and empathized with the user, offering greetings, and expressing sympathy for the user’s injury. Although both models consistently advised consulting a doctor, BARD conveyed a warmer, more welcoming tone. Conversely, Bing AI manifested the least personable demeanor, utilizing cold, clinical language and frequent third-person pronouns. Although Table 2 shows Bing AI outperforming BARD in the Flesch-Kincaid Grade Level and Coleman-Liau Index, this is due to BARD's failure to answer prompt 3, adversely affecting its scores. Noting that half of the comparisons were statistically insignificant, the authors suggest further investigations to acquire more robust results.

Deploying AI chatbots in clinical settings engenders ethical concerns, encompassing potential patient confidentiality breaches and inaccuracies due to error-prone public health data. Such issues may protract diagnosis, compromise patient safety, and entail legal consequences. To establish reliable and accountable AI systems, prioritizing transparency, explainability, and adherence to regulations and privacy policies is imperative. Addressing data and algorithmic biases and ensuring ongoing monitoring, evaluation, and ethical guideline adherence is essential for developers and users.

This study's principal limitation was its reliance on a specialized cohort of certified Plastic Surgeons and trainees to assess LLMs’ coherence, comprehensibility, and usability, which may impede generalizability and introduce subjectivity and biases. Nonetheless, this research constitutes a preliminary exploration, guiding future inquiries with varied clinician samples to appraise LLMs’ utility in healthcare settings.

This study showed that ChatGPT consistently provided more reliable, evidence-based clinical advice than BARD and Bing AI. However, LLMs generally lack depth and specificity, limiting their use in individualized decision-making. Healthcare professionals are crucial in interpreting and contextualizing LLM responses, especially for complex cases requiring multidisciplinary input. Future research should enhance LLM performance by incorporating specialized databases and expert knowledge, ensuring traceability and credibility of AI-generated content, and integrating LLMs with human expertise to advance nerve injury management and support patient-centered care.