The Knowledge of Large Language Models Regarding Response Evaluation Criteria in Solid Tumors: A Comparative Study with Prompt Effect

Introduction

Methods

The MCQs did not include any authentic patient data or images; therefore, ethical committee approval was neither required nor applicable for this study. Methodological transparency and reproducibility were ensured by adhering to the Standards for Reporting Diagnostic Accuracy Studies (STARD) guideline.[] An overview of the flowchart is presented in Figure 1.

Fig. 1. The workflow of the study.

Data Collection for Text-based and Case-based Multiple-choice Questions
A total of 50 text-based MCQs and 30 case-based MCQs were utilized in the study. These questions comprehensively covered the all sections of RECIST 1.1 and tested the application of the information therein. Each question was carefully constructed to focus on a single, specific, and critical concept relevant to radiological practice under this guideline. Each MCQ had 5 choices and only one choice was correct. A complete list of text-based and case-based MCQs and dataset of the study are available in the Appendix.

Design of Input-output Procedures for LLMs
The three different input prompts provided to the LLMs were: Prompt 1: "Act like a professor of radiology who has 30 years of experience in oncological imaging, especially with studies on RECIST 1.1. Give just the letter of the most correct choice of multiplechoice questions that I will ask you. Each question has only one correct answer." Prompt 2: "You are a senior academic radiologist. I have some questions about RECIST 1.1. I will ask you multiple-choice questions with a single correct answer. Provide only the letter of the most accurate choice for each." Prompt 3: "I have a few questions about RECIST 1.1 criteria. Some of them are text-based, and some of them are case-based questions. I will present you with multiple- choice questions, and each has only one correct answer. Please reply with the letter corresponding to the best choice only, without any explanation." These prompts were consistently employed across eight distinct platforms with default hyperparameters by R3 in February 2025: Claude 3 Opus and 3.5 Sonnet (https://claude.ai.com), ChatGPT-o1, ChatGPT-4o (https://chat.openai.com), Gemini 1.5 Pro (https:// gemini.google.com), Mistral Large 2 (https://mistral. ai), Llama 3.1 405B (https://metaai.com), and Perplexity Pro (https://perplexity.ai). In order to assess the consistency in the responses of the three different prompts within each model, the responses of each prompt and the model were evaluated carefully, and the prompt with the most successful responses for all models was recorded by R3.

The MCQs were administered sequentially within a single conversation session per LLM to maintain uniformity. None of the LLMs underwent additional pre-training or fine-tuning by the study authors, and no supplementary details that could potentially affect the study results were provided (Fig. 2). R3 reviewed the LLM responses and categorized them as correct (1) or incorrect (0).

Fig. 2. The example of chat session with ChatGPT-o1.

Radiologist Performance Evaluation
R1 and R2 independently answered the MCQs in a blinded manner in January 2025 using their personal computers. They completed text-based MCQs first, immediately followed by case-based MCQs without any interval. R3 separately evaluated their answers and categorized them as correct (1) or incorrect (0).

Statistical Analysis
The Kolmogorov-Smirnov test assessed data distribution. Descriptive statistics (minimum, maximum, median, interquartile range, percentages) were calculated. As the data were non-normally distributed, non-parametric tests were used. Consistency and performance across three prompts were evaluated with the Friedman and McNemar tests; the latter compared correct response rates between LLMs and radiologists. Chi-square tests assessed differences by question type. Bonferroni correction was applied for pairwise comparisons (p ≤0.028 significant), while p ≤0.05 indicated significance for consistency and prompt-related analyses.

Results

Table 1 The consistency of LLMs responses with different prompts (Prompt 1, Prompt 2 and Prompt 3)

With "Prompt 1", Claude 3.5 Sonnet demonstrated the highest accuracy at 83.3%, followed by R2 and Gemini 1.5 Pro, both of which achieved 80.0% (p>0.028). R1 closely followed with 76.7%, while ChatGPT-4o, Llama 3.1 405B, and Mistral Large 2 each recorded 73.3% (p>0.028). Claude 3 Opus and ChatGPT-o1 shared an accuracy of 66.7%. Perplexity Pro exhibited the lowest accuracy among LLMs and radiologists, with 60.0% (p>0.028) (Fig. 3).

Fig. 3. The accuracy of LLMs and radiologists on multiple choice questions.

There was no significant difference in accuracy on case-based MCQs among LLMs and between LLMs and radiologists (p>0.028) (Table 2).

Table 2 Comparison of the performance of LLMs and radiologist on case-based multiple-choice questions

Text-Based MCQs
Similar to case-based MCQ, all models reached the highest performance with "Prompt 1" among the three different prompts. The answers given by all models to the questions with these prompts were consistent, and there were no significant performance differences among models with them (p>0.05) (Table 1).

With "Prompt 1", Claude 3.5 Sonnet achieved the highest accuracy at 90.0%, followed by Claude 3 Opus and ChatGPT-o1, both scored 84.0% (p>0.028). ChatGPT-4o recorded an accuracy of 82.0%. Gemini 1.5 Pro attained 74.0%, the accuracy of Mistral Large 2 and Llama 3.1 405B at 72.0%. R2 (T.C.) had a slightly lower accuracy of 70.0% (p>0.028). Perplexity Pro demonstrated the lowest performance among all models, with an accuracy of 68.0% (Fig. 3).

Claude 3.5 Sonnet outperformed Mistral Large 2, Llama 3.1 405B, and Perplexity Pro, achieving the highest scores on text-based questions (p=0.012, p=0.022, p=0.007). It also demonstrated superior performance according to R1 and R2 (p=0.021, p=0.021).

When other LLMs were compared among themselves and with radiologists, there was no significant difference in performance between them (p>0.028) (Table 3).

Table 3 Comparison of the performance of LLMs and radiologist on text-based multiple-choice questions

Discussion

Coskun et al.[] evaluated ChatGPT using 59 prostate cancer questions from the European Urology Patient Information Society and reported suboptimal accuracy (mean: 3.62±0.49) requiring improvement. Similarly, Lombardo et al.[] tested ChatGPT (August 2023) with 195 questions from the EAU 2023 prostate cancer guidelines; expert review showed only 26% completely correct answers, with accuracy varying by section (best in follow-up/quality of life, poorest in diagnosis/ treatment. Similar to our results, in a recent study assessing LLMs in breast cancer care, three models- GPT-3.5, GPT-4, and Gemini (formerly Bard)-were evaluated using 60 MCQs covering treatment, diagnostic techniques, imaging interpretation, and pathology in breast cancer. GPT-4 achieved a 95% accuracy rate, outperforming GPT-3.5 (90%) and Gemini (80%), with statistically significant differences observed among the models (p=0.010). Furthermore, the models performed consistently across questions sourced from public databases and those formulated by radiologists.[] Also, Cao at al.[] evaluated LLMs in hepatocellular carcinoma diagnosis and management questions and found that ChatGPT-3.5, Gemini, and Bing answered only 45%, 60%, and 30% of basic clinical questions accurately, respectively, with even fewer responses deemed both accurate and reliable. Although there are still different results in the literature about the competence of LLMs in radiology, our results indicate that LLMs have quite a theoretical knowledge about RECIST 1.1.

Another important result of our study is that LLMs responded as well as radiologists on text-based and case-based MCQs that require analysis of the findings and data obtained. This result suggests that LLMs are successful in analyzing and reasoning texts such as radiology reports and providing the status of the disease (progression, stable, or regression) according to the RECIST guideline, which is the most critical for clinicians. To our best knowledge, there are no studies evaluating the performance of LLMs on case-based questions about cancer. Previous studies have evaluated LLMs" knowledge of cancer and cancer-related guidelines, which were only text-based. Çıtır reported that ChatGPT-3.5 gave largely correct answers to questions about oral cancer, 51.25% gave "very good" and 46.25% gave "good" answers, and the overall reliability was 97.5%.[] Similarly, Yurtcu et al.[21] demonstrated that ChatGPT has strong accuracy in answering frequently asked questions about cervical cancer. Beyond these studies, our study uniquely demonstrates that LLMs perform quite adequately on case-based MCQs in line with the correct analysis. With this finding, we believe that our study may be a leading point for further multicenter studies that evaluate the performance of LLMs in real-patient scenarios.

In our study, all LLMs with three different prompts showed great consistency with minor differences in responses. Due to the nature of LLMs, it is a surprising result that these models, which largely determine their answers according to the given prompt, perform similarly with different prompts to the questions about RECIST 1.1.[,] In contrast to our results, Russe et al.[] demonstrated the prompt effect, transforming a generic request into a precision prompt increased ChatGPT-4's factual correctness and decreased hallucinations, while a zero-shot chain-of-thought format further improved explain ability and user trust. Nguyen et al.[] tested ChatGPT mixed text-and-image multiple- choice questions from the 2022 ACR in-training examination; an "encouraging" prompt boosted overall accuracy to 61%, whereas a "threatening" prompt reduced accuracy to 48% and tripled the non-response rate. These studies suggest that when a task is openended or cognitively complex, the model"s probabilistic reasoning is highly sensitive to contextual cues embedded in the prompt. RECIST 1.1 evaluation, however, is a narrowly defined, rule-based exercise, so the knowledge retrieved is limited, and the model"s decision space is tightly constrained; once the required rules are invoked, rewording the prompt affords little additional leverage. Therefore, the prompting effect could diminish as the task approaches deterministic guideline application rather than broad clinical reasoning.

The impressive performance of Claude 3.5 Sonnet- achieving 83.3% accuracy on case-based MCQs and 90% on text-based MCQs-indicates great potential of its model in this field. The observed minor variations in accuracy among LLMs can be largely attributed to differences in their underlying architectures. Models with real-time web access capability, such as Gemini 1.5 Pro and Perplexity Pro, frequently derive their responses from non-scientific sources, which may account for the comparatively lower performance of web-enabled LLMs relative to those without internet access. In contrast, some of the ChatGPT and Claude models are trained on closed datasets, potentially contributing to their enhanced reliability.

Limitations of the Study
Our study has a few limitations. First, the number of questions was limited, and the assessment relied solely on MCQs. The performance of LLMs on open-ended questions was not evaluated in the study, which may have led to exaggerated LLM performances.

Second, we compared the accuracy of LLMs against two general radiologists with seven years of experience. It is likely that more experienced senior radiologists, particularly those with more specialized knowledge about oncological imaging, would achieve higher performance. Senior radiologists who are specialized and/ or subspecialized in this field may perform even better than LLMs, but since follow-up images of cancer patients are often evaluated by general radiologists in daily practice for many different reasons, the radiologists included in this study are general radiologists to better reflect real-life practice.

Lastly, this study assessed the performance of LLMs about RECIST 1.1 textually, while visual evaluation remains an integral component of radiological assessment. As such, the results of our study may not fully reflect the real-world applicability of LLMs in this field. It is important to emphasize that while LLMs performed well on structured MCQs, this could not directly reflect their ability in actual imaging interpretation for RECIST.

Conclusion

Informed Consent: The authors declare that this study was conducted without any commercial or financial relationships that could be construed as a potential conflict of interest.

Conflict of Interest Statement: The authors have no conflicts of interest to declare.

Funding: No funding was received for this study.

Use of AI for Writing Assistance: No AI technologies utilized.

Author Contributions: Concept - E.Ç.; Design - E.Ç., T.C.; Supervision - Y.C.G.; Materials - E.Ç.; Data collection and/ or processing - E.Ç.; Data analysis and/or interpretation - E.Ç., T.C.; Literature search - E.Ç., T.C.; Writing - E.Ç.; Critical review - E.Ç., T.C., Y.C.G.

Peer-review: Externally peer-reviewed.

References