bims-arines Biomed News
on AI in evidence synthesis
Issue of 2026–11–22
23 papers selected by
Farhad Shokraneh, Systematic Review Consultants LTD



  1. Laryngoscope. 2026 Nov 14.
       OBJECTIVES: The exponential growth in medical literature has strained traditional evidence synthesis approaches, prompting interest in artificial intelligence (AI) as a potential solution. Thus, we sought to assess the concordance and time savings between humans and AI in the systematic review process for a specialized otolaryngology topic.
    METHODS: A mock systematic review on immunotherapy for HPV+ oropharyngeal squamous cell carcinoma was conducted. Title/abstract and full-text were independently performed by human reviewers using Covidence and by Claude by Anthropic, model Sonnet 3.7. Concordance at each phase was assessed with sensitivity and specificity, and total time expenditure was documented for both approaches. Claude Sonnet 4.5 was subsequently applied to full-text screening of a published systematic review as proof-of-concept validation.
    RESULTS: Of the 1337 abstracts screened, human reviewers identified 43 for full-text review and ultimately included 17 manuscripts. Claude demonstrated an overall sensitivity of 17.6% (3/17) and specificity of 96.3% (1271/1320) when performing independent screening of all abstracts. When provided human-curated abstracts for full-text screening in an "AI-assisted" methodology, it achieved 100% (17/17) sensitivity and 57.7% (15/26) specificity. Validation against a published review achieved 100% sensitivity and 97.8% specificity. AI-driven screening required 325 active human minutes versus 1340 m for traditional human review, representing a theoretical active human time savings of over 75%.
    CONCLUSION: While AI cannot yet replace human judgment in systematic reviews, hybrid methodologies combining AI speed with human clinical expertise may represent the most promising approach for maintaining both efficiency and quality in evidence synthesis for rapidly evolving medical fields.
    LEVEL OF EVIDENCE: N/A.
    Keywords:  artificial intelligence; large language models; literature screening; otolaryngology; systematic review
    DOI:  https://doi.org/10.1002/lary.71037
  2. Clin Transl Sci. 2026 Nov;pii: e70780. [Epub ahead of print]19(11):
      The conventional systematic review is built around a finite cycle of literature searching, study selection, data extraction, critical appraisal, and synthesis. Although this process remains essential to evidence-based medicine, the resulting publication represents a snapshot of an evidence base that may continue to evolve. Advances in artificial intelligence are creating opportunities to accelerate individual review tasks and reconsider how evidence synthesis could be maintained and updated over time.
    Keywords:  algorithms; meta analysis; methodology; technology; translational medicine
    DOI:  https://doi.org/10.1111/cts.70780
  3. Ann Rheum Dis. 2026 Nov 16. pii: S0003-4967(26)00606-0. [Epub ahead of print]
    Working Group for EULAR AI-driven Screening
       OBJECTIVES: This study aims to improve the efficiency of title and abstract screening in systematic literature reviews (SLRs) supporting European Alliance of Associations for Rheumatology (EULAR) recommendations and points to consider, by developing and validating stopping criteria for active learning-assisted screening that achieve ≥95% recall (ie, sensitivity).
    METHODS: Simulation studies analysed 40 development and 5 validation datasets (202,177 records) from completed and ongoing EULAR SLRs, using active learning models (h3, u3, and u4) in ASReview. Time-to-event analyses estimated the proportion of records that required screening to reach 95% recall and the minimum safe streak length (longest sequence of irrelevant records before reaching 95% recall). Empirical stopping criteria were derived, tested on independent datasets and evaluated for workload reduction and time savings.
    RESULTS: Datasets averaged 4493 records (SD, 2851), with a mean of 67 (SD, 46) records included. The u4 model was optimal for balancing screening efficacy and computational efficiency. Target recall was achieved after screening 35% of records or reaching sequences of 15% nonrelevant records. A combined criterion (26% screened plus 10% irrelevant streak) provided an alternative stopping criterion. Validation met target recall. Screening workload was reduced by 67.1%.
    CONCLUSIONS: Empirically derived stopping criteria using an active learning tool can substantially accelerate SLR screening while maintaining high recall (≥95%), allowing reviewers to skip about two-thirds of records during title and abstract screening. Artificial intelligence-assisted screening can substantially reduce evidence synthesis workload, thus supporting more efficient development of EULAR recommendations and points to consider.
    DOI:  https://doi.org/10.1016/j.ard.2026.09.021
  4. Ann Epidemiol. 2026 Nov 14. pii: S1047-2797(26)00317-0. [Epub ahead of print]124 110319
       BACKGROUND AND OBJECTIVE: Rapid growth in medical literature limits traditional systematic review methods, particularly in high-volume fields like maternity care. This paper presents a novel methodological approach using a hybrid workflow that combines Large Language Model (LLM) semantic data extraction with programmatic filtering to automate title and abstract screening.
    METHODS: The workflow was deployed on a high-volume clinical dataset (51,281 records retrieved from six databases) as a validation case study. Google Gemini 1.5-pro and 2.5-flash extracted 12 structured semantic variables, followed by a seven-stage rule-based filtering pipeline to determine eligibility. The workflow's accuracy was validated against a large-scale manual audit.
    RESULTS: The workflow reduced the human manual screening burden by 96%. Validation demonstrated an estimated sensitivity of 89.9% (95% confidence interval; CI 87.3-92.1%), specificity of 96.4% (95% CI 96.2-96.5%), and false negative rate of 0.18% (95% CI 0.09-0.35%), achieving accuracy comparable to dual-human review. Total API expenditure was US$458.73. Subgroup analysis found no change in screening accuracy after changing LLM.
    DISCUSSION: This approach offers a scalable, transparent, cost-effective, and model-agnostic solution for evidence synthesis. By decoupling data extraction from inclusion logic, researchers maintain granular control over eligibility criteria, supporting scalable evidence synthesis and living systematic reviews.
    Keywords:  Abstract Screening; Automation; Bibliometrics; Caesarean Section; Evidence Synthesis; Large Language Models; Living Reviews; Long-term Health Effects; Maternal Health; Scoping review
    DOI:  https://doi.org/10.1016/j.annepidem.2026.110319
  5. BMC Med Res Methodol. 2026 Aug 27. pii: 256. [Epub ahead of print]26(1):
       BACKGROUND: Large language models (LLMs) can facilitate the labor-intensive screening of literature for systematic reviews and meta-analyses. Ensembling multiple LLMs to reach consensus on screening may allow locally deployed LLMs to achieve acceptable performance while reducing hallucination errors and maintaining privacy and reproducibility. The Delphi technique - in which panelists independently and anonymously explain their decisions, and then iteratively revise their judgements in response to aggregated responses from other panelists until reaching consensus - may provide a useful framework to formalize this process. To our knowledge, there are no published empirical evaluations on how changes in prompting impact the performance of automated screening when using multi-agent LLM ensembles in the medical domain.
    METHODS: We tested the impact of different prompts and workflow modifications on the performance of an automated abstract screening workflow that integrated five LLMs into a Delphi ensemble for complex classification tasks. Our test dataset was a published systematic review of biomarkers of non-syndromic autism spectrum disorder, in which human screeners deemed 1,655 abstracts potentially eligible ("target abstracts") among 4,745 identified by the search. We performed 12 types of experiments: five variations in the wording of the eligibility criteria section of the LLM prompt using moderate ("primary"), simplified, detailed, structured, or verbatim prompts, and seven modifications to the workflow's technical features (i.e., the temperature, embedding model and adapter, few-shot example sampler, safety net, and ensemble settings). Our primary evaluation metric was the workflow's recall (i.e., proportion of target abstracts detected by the workflow); secondary metrics included the work saved over sampling at 95% recall (WSS@95%, the effort saved by a screening system to identify 95% of the target abstracts compared to a random sampling strategy).
    RESULTS: Depending on the prompt and the workflow parameters, the screening system identified 59-99% of the 1,655 target abstracts. The WSS@95% ranged from 13 to 42%, indicating that changes in prompts and parameters moderately affect performance and efficiency. Simplified version of the prompt optimally balanced recall (98%) and WSS@95% (41-42%).
    CONCLUSION: Our study demonstrates the impact of prompt engineering and workflow modifications on an automated abstract screening workflow using Delphi-ensembled deployment of five open-weight LLMs. Future studies need to extend this work to automated data extraction to further advance the field of automated systematic review and meta-analysis.
    Keywords:  Consensus; Delphi; Document screening; Large language model; Psychiatry; Systematic review; Text embedding
    DOI:  https://doi.org/10.1186/s12874-026-02981-3
  6. JMIR Form Res. 2026 Nov 16. 10 e97813
       Background: Automation can be a useful efficiency tool for saving time in systematic reviews, but a trust gap and a steep learning curve persist concerning its accuracy and ease of use. It remains unknown whether these adoption barriers apply to the more complex process of systematic gray literature reviews (SGLRs), especially given the rapid evolution of large language models.
    Objective: This study seeks to identify the preferred features of SGLRs generated using an automation tool (automated SGLRs).
    Methods: Participants aged 18 years or older with experience using evidence from systematic reviews were recruited. A cross-sectional online Best-Worst Scaling Questionnaire was distributed through the authors' contacts, professional networks, and social media platforms. Descriptive statistics and multinomial logistic regression were used to estimate the baseline model for preferred features of automated SGLRs, while mixed logit analyses were used to explore subgroup heterogeneity.
    Results: A total of 168 respondents, comprising researchers, academicians, health care professionals, health policymakers, and postgraduates, completed the study. Multinomial logistic regression found that the top 3 preferred features for automated SGLRs were (1) high precision and sensitivity comparable to manual methods (best-worst [BW] score 269, and M-logit estimate B=2.26, odds ratio [OR] 9.60, 95% CI 7.72-11.93; P<.001), (2) acknowledgment and detailed explanation of the AI algorithms (BW score: 218, B=2.06, OR 7.85, 95% CI 6.35-9.7; P<.001), and (3) involvement of more than 2 independent human reviewers in the process (BW score: 118, B=1.54, OR 4.68, 95% CI 3.84-5.72; P<.001), when compared to the reference feature, which was the acceptance of automated SGLR output by peer colleagues. Subgroup analysis using a mixed logit model suggested that respondents aged 18 to 29 years exhibited the highest relative utility weights for detailed explanations (B=0.60, OR 1.81, 95% CI 1.29-2.55) and acceptance by everyone (B=0.35, OR 1.41, 95% CI 1.05-1.90) compared to other age groups. In contrast, academicians (B=1.36, OR 3.91, 95% CI 1.51-10.17) might prefer the detailed explanation feature of automated SGLRs more than those in other professions.
    Conclusions: The study concludes that automation tool developers should prioritize the accuracy of automated SGLRs, ensure algorithmic transparency, and integrate human-in-the-loop processes, as respondents most strongly preferred these features, which may support user trust.
    Keywords:  best-worst scaling; desirable features; gray literature; machine learning; systematic review
    DOI:  https://doi.org/10.2196/97813
  7. Qual Health Res. 2026 Nov 16. 10497323261487317
      The integration of large language models (LLMs) into qualitative health research is rapidly advancing, yet their methodological utility and limitations remain underexplored. This scoping review systematically maps current LLM applications in this field, evaluates their performance across analytical stages, and identifies future research directions and challenges. Following the PRISMA-ScR checklist and Arksey and O'Malley's framework, we systematically searched ten databases (Scopus, Cochrane Library, PubMed, Embase, CINAHL, Web of Science, China National Knowledge Infrastructure, Wanfang, China Biology Medicine, and VIP China Science and Technology Journal Database) for studies published between January 2017 and August 2025. Two independent reviewers screened records and extracted data, followed by descriptive synthesis and thematic analysis. From 8650 initial records, 17 empirical studies met the inclusion criteria. LLMs enhance qualitative analysis efficiency, with applications categorized into three dimensions: Qualitative Coding, Theme Generation, and Meaning-Making and Deep Interpretation. LLMs performed best in coding, automating repetitive tasks involving explicit content, and improving processing speed. However, their efficacy diminished in theme generation, where they struggled with category abstraction and lacked human-like logical rigor. In meaning-making, LLMs produced structured outputs but fell short in interpretive depth due to limited understanding of cultural nuances, metaphors, and contextual dependencies. This review highlights the promise of LLMs for optimizing qualitative analysis in health research, while acknowledging persistent challenges in interpreting nuanced sentiment, preventing hallucinations, and ensuring data ethics. We propose that LLMs currently serve as "augmentative tools" rather than "substitutes" for human researchers. Future research should prioritize model applicability in complex health contexts through human-in-the-loop approaches, alongside enhancing interpretative capacity and fostering reflexivity.
    Keywords:  data analysis; generative AI; health research; large language models; qualitative research; scoping review
    DOI:  https://doi.org/10.1177/10497323261487317
  8. Diabetologia. 2026 Nov 17.
      The aim of this EASD Guidelines Oversight Committee statement is to provide a framework for the responsible use of artificial intelligence (AI) tools in evidence syntheses supporting EASD clinical practice guidelines. The statement builds on the EASD Standard Operating Procedure for guideline development. It supports and operationalises the Responsible use of AI in Evidence Synthesis (RAISE) guidance within the framework of the Guidelines International Network (GIN) principles. The statement sets out EASD requirements for AI use in evidence synthesis, covering transparency, preplanning, additionality, credibility, ethics, accountability, compliance and evaluation. It also describes implementation steps within the EASD guideline process, including consideration of AI use, specification in evidence synthesis protocols, evaluation and approval, review-specific pilot testing, implementation oversight, and reporting. As such, it provides practical guidance on how AI can be responsibly incorporated into evidence syntheses for EASD guidelines, supporting transparent, validation-informed and human-supervised AI use.
    Keywords:  Artificial intelligence; Clinical practice guidelines; Evidence synthesis; Guideline methodology; Large language models; Standard operating procedure
    DOI:  https://doi.org/10.1007/s00125-026-06912-1
  9. NPJ Digit Med. 2026 Jul 31. pii: 909. [Epub ahead of print]9(1):
      Developing artificial intelligence (AI) for clinical research requires a comprehensive data foundation for model benchmarking and development. Here, we introduce TrialPanorama, a large-scale structured resource aggregating 1.6M clinical trial records from fifteen global registries linked with biomedical ontologies and literature. Using this resource, we construct 152K training and testing samples spanning eight clinical research tasks, including systematic review, trial design, and trial optimization. Benchmarking cutting-edge large language models (LLMs) reveals limited clinical reasoning capability in generic LLMs. In contrast, an 8B LLM developed on TrialPanorama using supervised fine-tuning and reinforcement learning outperforms 70B generic counterparts across all eight tasks, with relative improvements of 73.7, 67.6, 38.4, 37.8, 26.5, 20.7, 20.0, 18.1, and 5.2%, respectively. These results demonstrate the potential of domain-adapted AI to improve evidence synthesis and clinical trial design, establishing TrialPanorama as a foundation for scaling AI in clinical research.
    DOI:  https://doi.org/10.1038/s41746-026-02933-7
  10. Eur J Intern Med. 2026 Nov 16. pii: S0953-6205(26)00585-6. [Epub ahead of print] 107280
      
    Keywords:  AI; Accuracy; Artificial intelligence; Citation; CiteWell; Peer review; Performance; Reference; Research; Scientific publishing
    DOI:  https://doi.org/10.1016/j.ejim.2026.107280
  11. Front Oral Health. 2026 ;7 1937908
       Introduction: Generative AI tools (GAI) are widely used to speed literature reviews, yet their fidelity and reliability for literature synthesis remain unclear. This study had two Objectives: 1) summarize the evidence on associations between social determinants of health (SDOH) and periodontal disease (PD) in U.S. adults. 2) comparatively evaluate how accurately GAI tools summarize evidence compared to a PRISMA-guided human review.
    Methods: For the comparative analysis between a PRISMA-guided human systematic review and GAI tools, we selected the base models of ChatGPT-5.5, Gemini-3.5, and Perplexity. We conducted a search of five databases between January 2012 and December 2024, examining associations between SDOH and PD. A standardized prompt was used across all GAI tools. GAI output performance was evaluated using Checklist Coverage, Summary Fidelity, and Unsupported Content Score. Checklist Coverage assessed whether required checklist elements were included. Summary Fidelity Score assessed how well the outputs reflected the original content. Unsupported Content Score assessed whether the outputs introduced information not supported by the original study.
    Results: Seven studies met the inclusion criteria. ChatGPT-5.5 and Gemini-3.5 showed high checklist completeness (98%, 96%) with moderate content fidelity scores (71%, 70%). Perplexity showed checklist coverage (40%) and fidelity (24%). Unsupported content (ChatGPT-5.5 14%, Gemini-3.5 12%; Perplexity not evaluated due to low checklist coverage and summary fidelity) was mainly observed in the areas of limitations and future directions.
    Discussion/conclusions: GAI tools could help organize and generate quick, preliminary outputs, although they may not achieve the same level of rigor, consistency, and quality as a human-generated synthesis.
    Keywords:  comparative study; generative artificial intelligence; periodontitis; review literature as topic; social determinants of health
    DOI:  https://doi.org/10.3389/froh.2026.1937908
  12. Digit Health. 2026 Jan-Dec;12:12 20552076261493048
       Background: Generative AI is transforming health professions education, but AI hallucinations pose risks to academic integrity. Existing evidence is largely descriptive, leaving error mechanisms poorly understood.
    Objectives: This study evaluated AI hallucination prevalence, severity, and mechanisms across health professions education.
    Methods: Using a sequential explanatory mixed-methods design, twenty prompts (citation, factual, data-claim) were submitted to ChatGPT (GPT-5.2, free-tier), yielding 80 responses (four generations per prompt). Data were analyzed collectively. Two PhD-level reviewers with medical and nursing backgrounds assessed outputs using a structured hallucination checklist. Quantitative analyses (chi-square, Kruskal-Wallis) were integrated with qualitative thematic analysis of reviewer justifications.
    Results: Hallucinations occurred in 61.3% of responses; 48.9% of these were severe. Prevalence and severity were highest in citation-focused prompts (96.9%), followed by factual (62.5%) and data-claim prompts (12.5%) (χ2(2) = 41.16, p < .001; η2 = 0.63). Qualitative analysis revealed four mechanisms: fabricated/hybrid citations, bibliographic inconsistencies, unsupported numerical extrapolations, and conceptually inaccurate framework descriptions. Hallucinations frequently blended authentic and fabricated elements. Conclusions: AI hallucinations in health professions education are prevalent, domain-sensitive, and often severe. AI cannot function as an autonomous scholarly authority; structured verification and AI literacy training are essential.
    Keywords:  AI hallucinations; Artificial intelligence; ChatGPT; academic integrity; health professions education; large language models
    DOI:  https://doi.org/10.1177/20552076261493048
  13. Med Sci Educ. 2026 Dec;36(6): 2739-2745
      Generative Artificial Intelligence (GenAI) is now common in academic medicine. Yet, large language models (LLMs) can produce fabricated content and invented references that look genuine. Hallucination is inherent to how these models are trained and cannot be eliminated through better design alone; its frequency can be reduced and its detection improved. Drawing on daily practice in research and teaching, we propose eight practical considerations for reducing hallucination and an 8Rs model for structuring prompts. No single consideration is sufficient on its own. Applied collectively, under human oversight, they support the responsible use of GenAI in medical research and education.
    Keywords:  AI hallucination; Gen AI; Large language models; Medical education; Prompt engineering
    DOI:  https://doi.org/10.1007/s40670-026-02928-9
  14. J Clin Epidemiol. 2026 Nov 19. pii: S0895-4356(26)00470-1. [Epub ahead of print] 112594
       OBJECTIVE: Network meta-analysis (NMA) is increasingly used to compare multiple interventions and inform clinical guidelines, health policy, and decision-making. However, inconsistent reporting and challenges in communicating complex findings can limit the accessibility, transparency, and practical application of NMA results. To address these challenges, we developed the NMA Knowledge Interactive Translation (NMA-KIT) tool, an interactive web-based application integrated within NMAstudio 2.0, to enable a seamless workflow from analysis to communication of results.
    STUDY DESIGN AND SETTING: The tool was developed to enable a seamless workflow from analysis to communication of results and was demonstrated using an exemplar NMA of systemic treatments for psoriasis.
    RESULTS: NMA-KIT provides a structured and user-friendly framework for communicating NMA findings. The Detailed Report enables in-depth exploration of treatment effects, underlying data, assumptions, and certainty of evidence, whereas the KIT Summary presents concise, clinically focused results. Additional functionalities include interactive data filtering, customizable visualizations, absolute risk estimation, access to study-level information, and collaborative project sharing.
    CONCLUSION: NMA-KIT represents an important step forward for bridging the gap between NMA results and decision-making. By combining interactive visualization, flexible exploration, and integrated project sharing within NMAstudio 2.0, the tool has the potential to improve transparency, reproducibility, and communication, ultimately supporting more informed and collaborative evidence-based decisions.
    Keywords:  decision-making; evidence synthesis; interactive visualizations; knowledge translation; network meta-analysis; open-access software
    DOI:  https://doi.org/10.1016/j.jclinepi.2026.112594
  15. Genome Biol. 2026 Aug 22. pii: 363. [Epub ahead of print]27(1):
       BACKGROUND: Generative AI is increasingly used to extract structured information across domains, but its reliability in academic and clinical research, where precision and accuracy are essential, remains largely unexplored. This study evaluates the ability of Large Language Models based algorithms to generate accurate, literature-based summaries of human genomic variants, with a focus on real-world usability.
    RESULTS: We benchmark five open-access generative AI platforms-ChatGPT, MistralAI, VarChat, Perplexity, and ScholarAI-across 40 curated variants equally divided between somatic and germline settings. For each variant, summary reports are generated and blindly evaluated by domain experts using five defined metrics. VarChat emerges as the top-ranked tool, showing the highest summarization accuracy, citation relevance, and robustness against hallucinations. Gpt-4o consistently ranked second, showing particularly stable robustness in conditions where the literature was scarce. Perplexity and ScholarAI, despite being literature-focused, rank lowest across most metrics. Tool performance is strongly influenced by the availability of peer-reviewed literature, confirming that current generative models remain sensitive to data scarcity.
    CONCLUSIONS: Our findings highlight the heterogeneity of current generative AI tools in genomic variant interpretation workflows. While some platforms already provide useful outputs, reliable integration into basic and clinical research requires expert validation and domain-related fine-tuning. This work provides a first systematic expert-evaluation framework for benchmarking large language model-generated content in variant genomics and underscores the need for caution when using these tools to support variant interpretation.
    Keywords:  Benchmarking; Clinical genomics; Generative AI; Genomic variants; Germline mutations; Hallucinations; Large language models; Literature summarization; Somatic mutations; Variant annotation
    DOI:  https://doi.org/10.1186/s13059-026-04237-1
  16. Glob Health Action. 2026 Dec;19(1): 2761841
      Progress to end the global tuberculosis (TB) epidemic remains off track despite current efforts. Conferences generate emerging evidence to help achieve End TB 2035 targets, but these signals are rarely assessed for their alignment with End TB targets and for adoption readiness. Therefore, we aimed to identify and prioritize TB research from the Union World Conference on Lung Health (WCLH) 2025 against End TB 2035 targets, using an adoption readiness framework. We conducted an AI-assisted, human-verified synthesis of conference presentations. Speech recognition and three large language models supported transcription, structured extraction, verification, evidence classification, scoring and target mapping. Adoption readiness of research findings was assessed across development stage, affordability, scalability, implementation feasibility, financing, health-system compatibility and equity. Of 247 presentations included, 13 were classified as Tier 1, 128 as Tier 2 and 106 as Tier 3. Diagnostics and treatment showed the highest adoption potential (shortened regimens, pooled molecular testing and computer-aided radiography). Prevention had moderately strong evidence but lower adoption potential. Vaccines and prevention strategies carried the highest potential population impact but the lowest readiness, remaining uncertain to be ready before the mid-2030s. Major research gaps persisted for pregnancy, children, extrapulmonary TB, and in resource-limited settings. Despite meaningful scientific progress, current evidence was neither sufficiently mature nor equitably deployable to place the 2035 targets within reach. Greater investment, implementation evidence, health-system capacity and political action are required to translate innovation into global impact. AI-assisted, human-verified synthesis can rapidly organise emerging evidence, but its outputs are hypothesis-generating and require independent validation.
    Keywords:  End TB 2035 goals; Tuberculosis; adoption readiness; artificial intelligence; the Union WCLH 2025
    DOI:  https://doi.org/10.1080/16549716.2026.2761841
  17. medRxiv. 2026 Nov 13. pii: 2026.11.11.26367833. [Epub ahead of print]
       CLINICALTRIALS: gov provides valuable information for vaccine research, but fragmented data structures and inconsistent vaccine naming complicate systematic cross-trial analysis. The Study-Experiment-Assay Common Data Model (SEA-CDM) provides a standardized structure for biomedical study data, while biomedical ontologies supply semantic relationships among concepts. We extracted 9,147 vaccine-related study records from the Aggregate Analysis of ClinicalTrials.gov (AACT) database, mapped their records into SEA-CDM, and linked vaccine products and reported adverse events (AEs) to the Vaccine Ontology (VO) and Ontology of Adverse Events (OAE), respectively. Building on the existing Ontology-based SEA Network (OSEAN), we organized the integrated data into relational tables and a Neo4j knowledge graph while retaining links to original registrations. We evaluated three applications to assess semantic trial retrieval, context-aware AE ranking and trial similarity, and trial design recommendation. First, VO-based query expansion retrieved 41 candidate RNA vaccine trials compared with 16 using exact VO identifier matching. Second, a context-aware AE ranking model incorporating SEA-CDM features and VO-derived vaccine families improved recall@10 from 0.317 to 0.383 over a frequency-based baseline. TransE and CompGCN embeddings achieved Hits@10 of 0.950 and 0.946, respectively, for retrieving trials sharing the same VO identifier within disease-specific candidate sets. Third, an ontology-based clinical trial design recommendation model was developed using several machine learning methods under leave-one-Material-out evaluation across 23 vaccine classes, although performance varied across classes. These findings demonstrate how structural and semantic integration can support multiple clinical trial analyses using vaccine studies as a case study. This framework provides a foundation for extending ontology-supported data integration and analysis to other clinical research domains.
    DOI:  https://doi.org/10.64898/2026.11.11.26367833
  18. JMIR Form Res. 2026 Nov 17. 10 e101797
       Background: Simulation-based education is a cornerstone of health care training, with debriefing representing the critical phase of a learning process. However, assessing debriefing quality remains challenging, relying largely on expert judgment and structured tools such as the Debriefing Assessment for Simulation in Healthcare (DASH). The emergence of AI, particularly large language models (LLMs), offers new opportunities for the automated analysis of educational interactions.
    Objective: This study aimed to explore the feasibility and characteristics of AI-automated analysis of postsimulation debriefings using the DASH framework.
    Methods: We conducted a prospective, single-center, observational study at the Centre d'Apprentissage des Attitudes et Habiletés Cliniques (Université de Montréal). Postsimulation debriefings involving anesthesia residents in postgraduate years 2 and 4 were conducted by a single instructor, audio-recorded, transcribed, and anonymized. Transcripts were analyzed using a natural language processing model (ChatGPT) with a structured prompt aligned with the 6 DASH elements. For each debriefing, the model generated a structured qualitative evaluation, including a rating for each DASH element using the original 7-point DASH effectiveness scale and an identification of strengths and areas for improvement, supported by transcript excerpts. No comparison with a human DASH-trained evaluator was performed.
    Results: A total of 30 debriefings were included, with no exclusions or technical issues. A complete, structured AI-generated evaluation across all 6 DASH elements was obtained for all 30 (100%) debriefings, meeting the predefined feasibility outcome. "Establishes an engaging learning environment," "Maintains an engaging learning environment," "Provokes engaging discussion," and "Identifies and explores performance gaps" each had a median rating of 6 (IQR 6-7, 6-6, 6-7, and 6-6, respectively). "Structures the debriefing in an organized way" had the lowest median rating, at 4 (IQR 4-5), while "Helps trainees achieve or sustain good future performance" had a median rating of 5 (IQR 5-6). A cross-debriefing analysis identified recurring patterns across sessions, including consistent learner engagement and opportunities to improve debriefing structure and the formulation of strategies to support future performance.
    Conclusions: This study demonstrates the feasibility of AI-assisted qualitative assessment of postsimulation debriefings using an LLM prompted in accordance with the DASH framework. AI may represent a complementary tool to support pedagogical analysis, instructor reflection, and faculty development in simulation-based education. Although this exploratory study focused on textual transcript analysis without incorporating nonverbal communication or expert benchmarking, the findings support the feasibility of AI-assisted debriefing assessment and provide a foundation for future validation studies. Further research comparing AI-generated and expert evaluations could help refine these approaches and inform their integration into simulation-based education and faculty development.
    Keywords:   DASH framework ; AI; Debriefing Assessment for Simulation in Healthcare framework; artificial intelligence ; debriefing ; large language models ; medical education; simulation-based education
    DOI:  https://doi.org/10.2196/101797
  19. Gynecol Oncol Rep. 2026 Oct;67 102187
       Objectives: To evaluate the guideline concordance, accuracy, and completeness of unprompted ChatGPT-4 and Gemini responses to gynecologic oncology case vignettes benchmarked against NCCN guidelines across five clinical domains for patient education and medical training.
    Methods: Ten clinical vignettes encompassing five topics were proposed to reflect common yet complex decision-making scenarios in gynecologic oncology. Each vignette was posed without prompting in publicly-available versions of ChatGPT-4 and Gemini in 2025. Responses were independently scored by five board-certified gynecologic oncologists using a 0-2 scale. Scores were based on concordance with 2025 NCCN Guidelines. Descriptive statistics summarized performance patterns, and inter-rater reliability was measured using Gwet's AC1.
    Results: Across all domains, Gemini achieved a slightly higher overall concordance with NCCN guidelines than ChatGPT (mean score 1.82 vs. 1.62), with 82% and 64% of responses, respectively, rated as accurate and complete. There was one isolated instance of inaccurate (score of 0) output graded. Domain-specific analysis revealed that ChatGPT demonstrated a nonsignificant inferior average concordance compared to Gemini in all clinical categories. Both models showed lower completeness in survivorship and adjuvant treatment topics. Qualitative reviewer feedback highlighted clinically relevant omissions and interpretive nuances.
    Conclusions: Although neither ChatGPT nor Gemini consistently outperformed the other in specific clinical domains, Gemini's slightly higher average overall concordance with NCCN guidelines underscores its potential advantages over ChatGPT. AI may provide largely accurate, guideline concordant clinical reasoning across diverse gynecologic oncology scenarios. Variability and occasional inaccuracies underscore the need for clinical oversight before integration into care pathways or patient guidance.
    Keywords:  Artificial intelligence; Clinical vignette; Medical education; NCCN guidelines
    DOI:  https://doi.org/10.1016/j.gore.2026.102187
  20. Digit Health. 2026 Jan-Dec;12:12 20552076261500454
       Objective: General-purpose large language models (LLMs) show promise for medical question answering (QA) but face oncology-specific challenges, including hallucination, weak evidence attribution, and limited handling of sensitive patient queries. This study develops an evidence-grounded, patient-oriented oncology QA system integrating hybrid retrieval and specialised multi-agent processing to improve evidence attribution, safety, and empathetic communication.
    Methods: The proposed OncoAgentic-RAG is a collaborative multi-agent hybrid retrieval-augmented generation framework for oncology QA. Its knowledge base comprises standard oncology textbooks and clinical guidelines. Dense ChromaDB and BM25 sparse retrieval are combined for evidence retrieval and fusion in a LangGraph-based multi-agent architecture that incorporates crisis detection, intent orchestration, medical knowledge retrieval, treatment guidance, empathy-aware generation, clinical verification and patient-friendly response synthesis. The system was evaluated on the 1,015-question OncoIntentQA benchmark using retrieval, semantic, faithfulness, relevance, safety, completeness, empathy, and user-friendliness measures, with comparisons against NoRAG, BaselineRAG, and AgenticRAG. Generated responses were additionally assessed by clinical experts for safety and appropriateness.
    Results: OncoAgentic-RAG achieved Precision@5 of 0.8120, Recall@5 of 0.7980, BERTScore F1 of 0.7634, Faithfulness of 0.8146, and Answer Relevance of 0.8625. It achieved a weighted S.C.O.P.E. (Safety, Completeness, Originality, Precision, and Efficiency) score of 4.68/5.0, Empathy Composite of 0.9180, and User-Friendliness of 0.9580. Clinical expert assessment provided additional evaluation of response safety and content.
    Conclusions: Integrating hybrid retrieval with specialised multi-agent processing improved evidence-grounded, source-traceable, and patient-friendly oncology QA within the evaluated benchmark. The findings demonstrate the technical utility of OncoAgentic-RAG for patient-oriented oncology information, while prospective clinical and safety validation remains necessary.
    Keywords:  Agentic RAG; Clinical question answering; Empathetic healthcare; Knowledge graph; Large Language Model; Multi-agents; Oncology; Retrieval-Augmented Generation
    DOI:  https://doi.org/10.1177/20552076261500454
  21. Langenbecks Arch Surg. 2026 Nov 16. pii: 3. [Epub ahead of print]412(1):
       PURPOSE: Large language models may support guideline screening and multidisciplinary tumour board preparation in gastrointestinal oncology, but their performance in postoperative treatment assessment remains insufficiently characterized. To evaluate guideline concordance, tumour board concordance, reproducibility, and discordance patterns of ChatGPT-generated postoperative treatment recommendations in gastrointestinal cancer.
    METHODS: Single-centre exploratory cross-sectional study using retrospectively assembled clinical case vignettes assessed by a large language model (ChatGPT). Thirty adult patients with histologically confirmed pancreatic, gastric, or colorectal adenocarcinoma who underwent curative-intent surgery and subsequent multidisciplinary tumour board (MTB) review at a German tertiary referral centre (January 2025-January 2026) were included. Structured vignettes were entered into ChatGPT via a standardised prompt within a controlled guideline environment; each case underwent three independent runs (run-level) with majority voting to derive the decisive treatment recommendation (case-level). The primary outcome was concordance of ChatGPT recommendations with German guideline-based postoperative treatment recommendations on the case-level. Secondary outcomes included concordance with MTB recommendations, reproducibility, cross-guideline comparison, and qualitative characterisation of discordant recommendations (divided in minor or major deviation).
    RESULTS: ChatGPT-generated majority-vote recommendations were fully concordant with German guideline-based recommendations in 21 of 30 cases (70%). Concordance was higher in pancreatic and gastric cancer (8 of 10 cases [80%] each) than in colorectal cancer (5 of 10 cases [50%]). Multidisciplinary tumour board recommendations demonstrated the same overall concordance rate (21 of 30 cases [70%]). Concordance decreased in clinically advanced disease stages. Cross-classification demonstrated complete guideline concordance in both ChatGPT-generated and tumour board recommendations in 17 of 30 cases (56.7%), whereas major LLM discordance despite guideline-adherent tumour board recommendations was uncommon (1 of 30 cases [3.3%]). In exploratory run-level analysis, 23 of 90 outputs (25.6%) were discordant, comprising 14 (15.6%) major and 9 (10.0%) minor deviations. Complete agreement across repeated runs occurred in 61 of 90 case-guideline combinations (67.8%).
    CONCLUSION: ChatGPT-generated treatment recommendations showed relevant but variable concordance with guideline-based and multidisciplinary tumour board recommendations. Discordance clustered in clinically complex treatment situations, supporting a possible role for LLM-based systems in guideline screening and tumour board preparation while requiring expert physician oversight.
    TRIAL REGISTRATION: ClinicalTrals.gov: #NCT07592338.
    Keywords:  Antineoplastic Protocols; Artificial Intelligence; Clinical Decision Support Systems; Neoplasms; Postoperative Care; Practice Guideline
    DOI:  https://doi.org/10.1007/s00423-026-04351-5