bims-arines Biomed News
on AI in evidence synthesis
Issue of 2026–10–25
23 papers selected by
Farhad Shokraneh, Systematic Review Consultants LTD



  1. J Health Econ Outcomes Res. 2026 ;13(2): 148-160
       Background: Artificial intelligence (AI) is being introduced into every stage of the health economics and outcomes research (HEOR) production line, from literature screening to causal estimation. The field lacks a defensible count of what has been done, and bibliometric claims about AI adoption circulate without measured error rates.
    Methods: Following PRISMA-ScR, we searched PubMed for AI terms co-occurring with HEOR terms for publications from 2010 through 2025, retrieved 2642 records, and included 2476 after screening. Records were classified into 8 nonexclusive application domains and 1 of 6 method categories using a published term dictionary. To quantify classification error, an independent large language model classifier, blinded to the dictionary labels and given only title and abstract, reclassified a random sample of 300 included records. We report per label counts, agreement, Cohen kappa, prevalence-adjusted bias-adjusted kappa, precision, recall, F1, and a precision-adjusted corpus size with exact binomial intervals. We separately assembled the primary performance evidence for each HEOR workflow, requiring a quantitative metric from a primary study.
    Results: Included publications rose from 29 in 2010 to 852 in 2025, a factor of 29.4, with the inflection at 2018 to 2019 rather than at the release of ChatGPT. AI reached 6.0% of all indexed HEOR output in 2025 while HEOR fell to 0.86% of all indexed medical AI output. Economic evaluation was the most frequently assigned domain (1559/2476, 63.0%), followed by real-world data and causal inference (496, 20.0%) and governance, reporting, and validity (494, 20.0%). Generative AI appeared in no record before 2021 and in 103 in 2025. Only 154 of 300 sampled records (51.3%; 95% confidence interval [CI], 45.7-56.9) were judged genuinely HEOR relevant, implying a precision-adjusted corpus of 1271 studies (95% CI 1131-1410). Agreement was substantial for real-world data and causal inference (kappa, 0.729) but negligible for economic evaluation (kappa, 0.144), where dictionary precision was 29.6%. Of 2476 records, 38 (1.5%) were randomized trials and 17 (0.7%) validation studies. The replication package is publicly available.
    Conclusions: The literature is growing faster than its own measurement infrastructure. Measured gains concentrate in tasks with observable ground truth, and predictive performance repeatedly fails to predict evidentiary performance.
    Keywords:  artificial intelligence; causal inference; health economics outcomes research; health technology assessment; machine learning; real-world evidence
    DOI:  https://doi.org/10.36469/001c.171937
  2. J Health Econ Outcomes Res. 2026 ;13(2): 161-177
       Background: Part 1 of this series found that measured gains from artificial intelligence (AI) in health economics and outcomes research (HEOR) concentrate in tasks with observable ground truth and disappear where the quantity of interest is a counterfactual. What regulators, health technology assessment (HTA) bodies, and sponsors have said about that division has not been assembled in one place.
    Methods: We assembled the primary regulatory, HTA, industry, and journal policy record through September 2026, requiring for each claim a named primary source and, where a quantity was reported, its provenance and whether the reporting party had a commercial interest. We downloaded the Food and Drug Administration (FDA) AI-enabled medical device list in full, tabulated published economic evaluations of AI, and set reported industry efficiency gains against disclosed use in submissions. We then evaluated the field against the task-based productivity framework of Nobel laureate Daron Acemoglu and derived a reporting standard from the documented failure modes.
    Results: The FDA has authorized 1614 AI-enabled devices, 1230 (76.2%) in radiology, rising from 18 in 2016 to 335 in 2025, and has received more than 500 drug and biologic submissions containing AI components since 2016; its January 2025 credibility assessment guidance remains a draft and excludes discovery and operational uses. Industry sources report time savings of 40% to 97% in regulatory writing and approximately 80% in payer dossier drafting, but every one of these figures comes from a single-arm comparison without a concurrent control and was reported by the company that deployed or sold the tool. AI use was disclosed in 3 of 1391 HTA documents (0.2%) and 0.6% of one journal's submissions, against surveys reporting use between 25% and 81%. Nineteen economic evaluations of AI have been published against 1614 authorized devices. No regulator or HTA body has been documented accepting or rejecting an AI-derived endpoint as such.
    Conclusions: Current disclosure instruments ask whether AI wrote the text. Reviewers need to know whether AI selected the evidence, defined the cohort, derived the endpoint, or generated the code, and whether the recommendation would change without it.
    Keywords:  artificial intelligence; health economics and outcomes research; health technology assessment; large language models; real-world evidence; regulatory science; reporting standards
    DOI:  https://doi.org/10.36469/001c.171942
  3. Environ Sci Technol. 2026 Sep 15. 60(36): 25110-25127
      Environmental evidence synthesis (EES) transforms diverse environmental evidence into trustworthy and decision-relevant knowledge for environmental governance and policymaking. Yet, the accelerating growth and heterogeneity of environmental literature are increasingly exceeding the capacity of conventional manual workflows. Here, we conducted an expert-led systematic review following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidance to assess the capabilities, reliability boundaries, and real-world applications of artificial intelligence (AI)-assisted EESs. From 38,840 records, we included 459 peer-reviewed studies and organized them into three evidence streams. Capability evidence revealed a substantial but weakly EES-adapted AI tool ecosystem with 158 directly usable tools but only three directly relevant to EES, indicating that environment-specific capabilities remain scarce and task-localized. Reliability evidence showed task-contingent AI performance, with more stable outputs for structured front-end tasks but persistent verification needs for judgment-intensive tasks. Real-world evidence shows that AI has already entered evidence synthesis practice, but its use in EES is advancing ahead of domain-specific validation. Together, these findings indicate that AI-assisted EES is moving from possibility to practice but not yet from adoption to trust. This transition will require trustworthy AI-assisted EES infrastructure, including task-specific environmental tools, expert-annotated benchmarks, transparent reporting and validation standards, and living EES systems for continuous updates. Only within such an infrastructure can AI move beyond a tool for efficiency gains and become a governed, credible capability for environmental knowledge production.
    Keywords:  artificial intelligence; benchmark evaluation; environmental evidence synthesis; evidence synthesis automation; living evidence system; systematic review
    DOI:  https://doi.org/10.1021/acs.est.6c09347
  4. Int J Technol Assess Health Care. 2026 Oct 19. 42(1): e95
      The use of artificial intelligence (AI) in health technology assessment (HTA) is rapidly emerging and evolving worldwide (1;2). While many may tout the potential of AI to enhance efficiency, accuracy, and speed of HTA, these technologies also present risks, concerns, and practical challenges that may limit responsible, ethical, and effective deployment. The 2025 HTAi Global Policy Forum (GPF) brought forward key themes regarding the use of AI in HTA, such as trust, human agency, and risk-based approaches (3). Others have also discussed the policy considerations of AI in HTA (4;5).
    DOI:  https://doi.org/10.1017/S0266462326104292
  5. J Med Libr Assoc. 2026 Oct 01. 114(4): 359-370
       Objective: This investigation compared six AI chat tools (ChatGPT, Claude, Copilot, Gemini, OpenEvidence, and Perplexity) to each other and to PubMed to assess their utility in medical literature searches.
    Methods: Each tool was prompted with two standardized prompts simulating an academic and a clinical literature need. Prompts were repeated three times per tool, and results recorded for each prompt iteration.
    Results: Main findings include the range in number of real and hallucinated articles for each AI tool, from no hallucinations in OpenEvidence to more than 80% hallucinated citations in ChatGPT. Across all tools, most real articles were assessed as relevant. Continued prompting of all tools produced duplicate results. Most tools showed very little consistency in citations provided to different researchers, suggesting low probability of replicating search results.
    Conclusions: Though limited by small sample size, inconsistent data, and the rapid pace of change in LLM technology, this study suggests some advantages and drawbacks of using AI tools for literature searching compared with PubMed. AI tools with few or no hallucinations and high relevance, such as OpenEvidence and Perplexity, performed well in terms of simply supplying a few article citations on the topic, but small results sets and lack of consistency and transparency limit their value for thorough research. These results suggest that AI tools are not presently a viable replacement for PubMed or other academic databases for systematic reviews or other in-depth literature searches but may be adequate for limited information needs or as a complement to database or web searches.
    Keywords:  AI; comparison; literature searches
    DOI:  https://doi.org/10.5195/jmla.2026.2284
  6. JSES Int. 2026 Nov;10(6): 101790
       Background: As the use of artificial intelligence (AI) expands into orthopedic surgery, the potential to use AI for review and summation of literature leaves researchers and clinicians questioning the quality of AI-driven literature reviews. This study aims to compare AI-generated literature reviews to those done by orthopedic surgery trainees.
    Methods: ChatGPT-4o and The-Literature, a directed AI platform, were asked to produce a literature review on 5 shoulder and elbow-related research topics. The primary output produced by ChatGPT-4, The-Literature, a fourth-year medical student, a third-year orthopedic surgery resident, and an orthopedic surgery sports medicine fellow were blinded then graded on a 1-5 Likert Scale for evidence and analysis, conclusion, references, and overall quality by 2 fellowship-trained orthopedic shoulder and elbow surgeons.
    Results: The average overall Likert ratings were 2.90 ± 0.88 for the medical student, 3.90 ± 0.74 for the resident, 3.50 ± 0.53 for the fellow, 2.30 ± 1.16 for ChatGPT-4, and 3.40 ± 0.84 for The-Literature. The resident and fellow reviews were significantly better than ChatGPT-4 (P = .0004, P = .0209), while the medical student review was not (P = .5092). None of the trainee reviews were significantly better than The-Literature. The-Literature's reviews were significantly better than ChatGPT-4 (P = .0412). The majority (58.8%) of ChatGPT-4's references were fabricated.
    Conclusion: A limited and directed AI search program was comparable to orthopedic surgery trainees and performed better than an open AI large language model. ChatGPT-4 had poor overall utility stemming mainly from its tendency to fabricate references.
    Keywords:  AI; Artificial intelligence; ChatGPT; Elbow; Literature review; Shoulder; The-literature
    DOI:  https://doi.org/10.1016/j.jseint.2026.101790
  7. J Med Libr Assoc. 2026 Oct 01. 114(4): 508-511
      We previously found that automated indexing in PubMed and Medline increased the risk of inappropriately excluding human studies when using the human studies filter from the Cochrane Highly Sensitive Search Strategy (HSSS). Following the National Library of Medicine's transition from the Medical Text Indexer-Auto (MTIA) to the Medical Text Indexer-Next Generation (MTIX) in 2024, we wanted to assess whether that was still the case. Using the same approach as our previous study, we reviewed a set of MTIX-indexed records that were excluded by the HSSS. Only 1.1% turned out to be human studies that were inappropriately excluded - a major improvement over not only MTIA but also human indexers in our previous comparison. However, problems remain with automated indexing and search filters. This commentary outlines our findings and our ongoing concerns.
    Keywords:  Automated MeSH indexing; commentary; filter
    DOI:  https://doi.org/10.5195/jmla.2026.2377
  8. Res Synth Methods. 2026 Oct 19. 1-40
      This study quantifies how prompting strategies interact with large language models (LLMs) to automate the screening stage of systematic literature reviews (SLRs). We evaluate six LLMs (GPT-4o, GPT-4o-mini, DeepSeek-Chat-V3, Gemini-2.5-Flash, Claude-3.5-Haiku, and Llama-4-Maverick) under five prompt types (zero-shot, few-shot, chain-of-thought [CoT], CoT-few-shot, and self-reflection) across relevance classification and six Level-2 tasks, using accuracy, precision, recall, and F1. Results show pronounced model-prompt interaction effects within this SLR-automation corpus: zero-shot maximizes recall for high-sensitivity relevance screening; CoT-few-shot yields stronger precision and F1; and self-reflection underperforms due to over-inclusivity and instability across models. GPT-4o and DeepSeek provide robust overall performance, while GPT-4o-mini performs competitively at a substantially lower dollar cost. A cost-performance analysis for relevance classification (per 1,000 abstracts) reveals large absolute differences among model-prompt pairings; GPT-4o-Mini remains low-cost across prompts, and structured prompts (CoT/CoT-few-shot) on GPT-4o-Mini offer attractive F1 at a small incremental cost. The cost-performance analysis also suggests that future screening workflows could incorporate staged, cost-aware procedures: (1) using lower-cost, high-recall configurations for initial screening and (2) reserving higher-cost or more precision-oriented configurations for later triage or uncertain records. These findings highlight LLMs' uneven but promising potential to support screening within corpora similar to the SLR-automation corpus examined in this study. Further validation is needed before generalising these results to broader systematic review screening tasks. By systematically analyzing prompt-model interactions, we provide a comparative benchmark and practical guidance for task-adaptive LLM deployment.
    Keywords:  LLM-based SLR automation; automated literature screening; evidence synthesis; model–prompt interaction; prompt engineering; systematic literature review (SLR)
    DOI:  https://doi.org/10.1017/rsm.2026.10110
  9. JAMIA Open. 2026 Oct;9(5): ooag210
       Objectives: To evaluate performance and efficiency of a lightweight open-source reasoning large language model (LLM) against an expert-derived reference standard for screening scientific publication databases.
    Materials and Methods: Two expert reviewers screened subsets of database records across 3 stages: (1) base eligibility (n = 200), (2) obesity focus (n = 200), and (3) women of reproductive age focus (n = 150). The LLM (gpt-oss-20b) screened all records (n = 20 118). Performance (F1-score and sensitivity [recall]) and efficiency (average screening rate) were assessed.
    Results: Interreader agreement was near-perfect ( κ : 0.99-1.00; all P < .001). LLM performance was high: F1-score, 0.83-0.94; sensitivity, 90%-95%; specificity, 86%-99%; accuracy, 89%-94%; precision, 77%-99%; and negative predictive value (NPV), 89%-96%. LLM screening was faster than reviewers (19.4 vs ∼1.0 records/min).
    Discussion: The LLM achieved high performance and efficiency relative to an expert-derived reference standard for screening.
    Conclusion: The use of lightweight open-source reasoning LLMs shows promise to support screening in evidence synthesis workflows.
    Keywords:  artificial intelligence; evidence synthesis; health equity; large language models; natural language processing; systematic literature review
    DOI:  https://doi.org/10.1093/jamiaopen/ooag210
  10. Oncology. 2026 Oct 20. 1
      Background The rapid growth of biomedical publishing in radiation oncology has created a need for automated methods to extract structured information from unstructured text. Large language models (LLMs) show promise in biomedical natural language processing, but their use for clinical trial data curation remains to be systematically evaluated. Methods We evaluated an LLM-based extraction system for radiation oncology trials on ClinicalTrials.gov. Forty-five classification questions spanning variables such as trial design, tumor type, stage, radiotherapy modality, systemic therapy, and outcomes were implemented using the data-element-extractor Python library with three open-source LLMs (GPT-OSS-20B, GPT-OSS-120B, and Qwen-3.6-27B). A manually curated ground-truth of 94 trials was used for performance evaluation based on accuracy, precision, recall, and F1-score. Results Qwen-3.6-27B achieved the highest overall accuracy of 92.0% across all questions. For the 39 Boolean questions, it demonstrated 93.0% accuracy, 81.6% precision, 83.7% recall, and an F1-score of 82.6%, with 23 Boolean questions achieving an F1-score ≥80.0%. Three numerical questions reached accuracy ≥80.0%. Applied to the full dataset, the system characterized the clinical landscape of radiotherapy trials, detailing the prevalence of advanced techniques (e.g., 13.7% SBRT) and the integration of systemic agents, including chemotherapy (57.3%) and immunotherapy (26.1%). Conclusion LLM-based extraction can enable accurate, scalable curation of radiotherapy trial data. While challenges remain with complex categorical or numerical variables, open-source LLMs can serve as effective components of transparent, auditable clinical research pipelines.
    DOI:  https://doi.org/10.1159/ocl/aebag024
  11. Bioinform Adv. 2026 ;6(1): vbag283
       Background: Collaborative Research Centers rely on FAIR-compliant, richly structured metadata, yet manual annotation is a major bottleneck. We implemented a search-augmented large language model (LLM) workflow within a local research data management system to pre-annotate biomedical entities, using human-in-the-loop verification to ensure data quality.
    Methods: The pipeline uses Gemini 3 Pro in a two-step prompting strategy: (1) identify dataset deposits and stable identifiers in articles converted to Markdown; (2) extract structured fields from curated repository landing pages rendered via a headless browser. To handle a highly hierarchical metadata schema, we flattened the schema for prompting and remapped outputs to strict JSON with granular provenance tags. Authors received pre-filled metadata and could accept, edit, or delete entries (TP, FP, FN mapping). Performance metrics (precision, recall, F1) were estimated as proportions and synthesized via random-effects meta-analysis.
    Results: Among 51 screened articles, the LLM identified a repository deposit in 31; authors responded for 17 (55% of articles with an identified deposit; 33% of all screened articles), yielding 39 verified datasets. False positives were rare (mean 0.23) and false negatives low (mean 1.46). Precision was consistently high at 99.65% (95% CI 98.42%-100.00%). Recall showed more variability at 93.75% (95% CI 89.79%-96.96%), and the F1 score reached 96.17% (95% CI 93.78%-98.11%).
    Conclusions: The hybrid workflow achieved high author-verified precision with moderate recall, shifting effort from drafting to reviewing while maintaining schema compliance. As these metrics rely on author acceptance and response rates were modest, broader, multi-site validation is needed.
    DOI:  https://doi.org/10.1093/bioadv/vbag283
  12. J Biomed Inform. 2026 Oct 23. pii: S1532-0464(26)00140-1. [Epub ahead of print] 105116
       OBJECTIVE: Descriptive scientific metadata in public repositories are often incomplete and inconsistent with community standards and ontologies, limiting data FAIRness. Large language models (LLMs) offer a promising approach to automatically standardizing such metadata when the LLM is provided with knowledge of the relevant standards in machine-actionable form (e.g., as metadata templates from systems such as the CEDAR Workbench). Entry of metadata standards through prompt engineering offers only fixed snapshots of the relevant standards, however, and it relies on an LLM's pretrained knowledge to interpret and satisfy constraints related to the standard. We evaluate whether giving an LLM access to metadata specifications and authoritative terminology at runtime improves automated metadata standardization.
    METHODS: We present ARMS, a tool-augmented LLM agent that retrieves complete CEDAR metadata templates and that queries authoritative biomedical terminology services dynamically at execution time. We compared ARMS with a prompt-based approach on 839 legacy metadata records describing experiment datasets from the Human BioMolecular Atlas Program (HuBMAP) with expert-standardized records serving as the reference standard.
    RESULTS: Across the corpus, ARMS outperformed the prompt-based approach in converting legacy metadata to standards-compliant form, increasing precision from 0.56 to 0.93 and recall from 0.51 to 0.85, with improvements across all field categories and assay types. The largest gains occurred for ontology-constrained fields, where precision increased from 0.36 to 0.92.
    CONCLUSION: Our findings show that access to appropriate metadata standards can substantially improve LLM-based standardization of legacy metadata. ARMS improves metadata standardization by giving LLMs runtime access to the authoritative resources that define valid metadata. Representation of metadata standards in machine-actionable form enhances the task of rectifying legacy metadata using LLM technology, especially when those standards can be queried dynamically.
    Keywords:  Data reuse; Large language models; Metadata standardization; Metadata templates; Ontologies; Software agents
    DOI:  https://doi.org/10.1016/j.jbi.2026.105116
  13. JMIR Form Res. 2026 Oct 20. 10 e100769
       BACKGROUND: Clinical trial data pipelines must follow Clinical Data Interchange Standards Consortium (CDISC) standards, including the study data tabulation model (SDTM) and analysis data model (ADaM). General-purpose large language models (LLMs) can produce plausible artifacts that omit required variables, misuse terminology, or invent derivation logic. Curated context may reduce these errors, but the value of organizing that context by workflow topology has not been separated from corpus access alone.
    OBJECTIVE: This study aimed to examine when workflow topology-guided skill loading adds value beyond a domain prompt, nonrouted curated context, or lexical retrieval, and whether bounded scheduling retains quality with less injected context.
    METHODS: A 45-node, 7-layer directed acyclic workflow graph was linked to 59 modular rule files (SKILL.md). A sequential exploratory evaluation examined 4 evidence regimes spanning domain-context, component isolation, lexically reachable, and topology-dependent conditions (10 tasks, 90 runs; 10 tasks, 180 and 120 runs; 24 tasks, 288 runs; and 18 tasks, 324 runs; the task, not the agent run, was the inferential unit). End points included LLM rubric scores, deterministic atomic criteria, required-evidence recall, and manifest checks. A blinded practicing statistical programmer independently rescored 60 outputs. Task-level paired inference used bootstrap 95% CIs, exact sign-flip tests, and Holm adjustment.
    RESULTS: Routed context scored 4.63 versus 4.20 for prompt-only (difference 0.43 on a 1-5 scale; 95% CI 0.25-0.63; Holm-adjusted P=.01) but did not exceed flat or random loading. The lower-context scheduler used 28.82% less context while retaining quality; all 559 injected fragments had recorded sources, and all 120 manifests were reproduced exactly. On lexically reachable tasks, graph+Best Match 25 (BM25) and BM25-only were equivalent within a 0.05 margin (difference +0.009; 95% CI -0.028 to 0.046; P=.67), where BM25 already recalled 95.8% of required evidence. On topology-dependent stress tasks, graph-only selection outperformed BM25-only (difference 0.348; 95% CI 0.178-0.530; nominal P=.002; dz=0.89) and raised required-evidence recall from 0.500 to 1.000. Expert-judge agreement was substantial for terminology (weighted κ=0.61) and moderate overall (intraclass correlation coefficient=0.60).
    CONCLUSIONS: Workflow topology serves as a conditional retrieval prior. It adds little when lexical retrieval already reaches the required evidence but recovers dependency-linked evidence when task wording does not identify it lexically. These findings, from a synthetic benchmark using one generator backbone and one expert rater, support topology-guided selection as a promising approach for controlled context assembly and motivate independent factorial validation.
    Keywords:  AI; CDISC; agentic AI; clinical data interchange standards consortium; clinical trials; context engineering; domain adaptation; knowledge graph; large language model; retrieval
    DOI:  https://doi.org/10.2196/100769
  14. Environ Sci Technol. 2026 Oct 06. 60(39): 27792-27802
      Life-cycle inventory (LCI) construction remains labor-intensive because relevant information is distributed across heterogeneous text, tables, and figures and is reported using inconsistent terminology, units, and system boundaries. Although large language models (LLMs) offer opportunities for automated information extraction, LCI-specific workflows that reliably convert literature-reported information to structured and traceable inventory records remain limited. Here, we present LCIMiner, an automated LLM-based workflow integrating literature retrieval and screening, text-based selection of LCI-relevant figures and tables, multimodal information extraction, task-specific self-review, and structured LCI construction without model retraining. We evaluated LCIMiner in aquaculture, polymer production, and biofuel production, which differ substantially in their production processes, inventory structures, and reporting conventions. Across these domains, LCIMiner achieved higher end-to-end extraction performance than matched single-model baselines and converted heterogeneous literature evidence to machine-readable, source-traceable LCI records. We further applied a separate human-guided harmonization procedure to organize the extracted aquaculture records into a cross-study analytical data set while preserving the original source information. Together, the automated extraction workflow and subsequent human-guided harmonization provide a transparent and extensible approach for developing reusable foreground LCI data sets from heterogeneous scientific literature.
    Keywords:  LCI automation; Life-cycle assessment; Literature mining; Multimodal information processing; Structured data extraction
    DOI:  https://doi.org/10.1021/acs.est.6c12624
  15. Behav Res Methods. 2026 Oct 19. pii: 329. [Epub ahead of print]58(11):
      Large language models (LLMs) are increasingly used to support qualitative research, yet clear methodological frameworks for integrating them into thematic analysis remain limited. A central challenge is how to scale qualitative coding with LLM assistance while preserving methodological rigor and reproducibility. This tutorial presents a structured, human-in-the-loop workflow for conducting large-scale thematic analysis with LLM support. Grounded in inductive and hybrid approaches to thematic analysis (Braun & Clarke, 2006), the workflow proceeds in three phases. First, researchers conduct manual thematic analysis on a subset of data to establish an initial codebook and assess relevance. Second, the same subset is analyzed using an LLM to iteratively refine prompts and evaluate alignment between human and model outputs. Third, the validated approach is applied to the full corpus using a modular four-step pipeline: key-idea extraction, chunk-level thematic coding, theme consolidation, and clustering, with iterative human verification through random sampling of outputs. We demonstrate the workflow using 2,952 publicly available Reddit posts. Structured prompt design and JSON-formatted outputs enable traceability between themes and source data and support reproducible analysis. Prompt engineering is treated as an iterative analytic process rather than a one-time instruction. Under controlled conditions, human and LLM coding showed high agreement (Cohen's κ = 0.91). All prompts, code, and reproducibility materials are openly available via an Open Science Framework repository. By providing detailed implementation guidance, embedded validation procedures, and safeguards against common failure modes such as hallucination and overgeneralization, this tutorial offers a practical and extensible framework for integrating LLMs into qualitative research across domains.
    Keywords:  Human-in-the-loop; Large language models; Large-scale text analysis; Prompt engineering; Qualitative research; Reproducibility; Thematic analysis
    DOI:  https://doi.org/10.3758/s13428-026-03207-6
  16. Arthritis Care Res (Hoboken). 2026 Oct 20.
       OBJECTIVE: Our objective was to evaluate the ability of different large language models (LLMs) to translate and analyze qualitative interviews describing pregnancy experiences among women with systemic lupus erythematosus (SLE). An additional objective was to compare themes identified by LLMs with those identified by human researchers.
    METHODS: We analyzed 26 semi-structured interviews originally conducted in Portuguese by Rodrigues et al. among women with SLE exploring their perinatal care experiences. Three prompting strategies (zero-shot, few-shot, and chain-of-thought) were used on the transcripts across four state-of-the-art LLMs (GPT-4o, Claude Sonnet 4.5, Gemini 2.5 Flash, and DeepSeek R) to generate key themes. Model outputs were evaluated using an LLM-as-a-judge framework assessing thematic coverage, thematic unity, and semantic similarity.
    RESULTS: Across models and prompting strategies, LLMs generated similar themes that closely aligned with those identified by human researchers. LLM-as-a-judge evaluations demonstrated high thematic unity and semantic similarity. Claude Sonnet 4.5 showed the most consistent performance. Few-shot prompting consistently outperformed zero-shot and chain-of-thought approaches.
    CONCLUSION: LLMs show promise as supportive tools for analyzing qualitative research by preserving thematic coherence and capturing the complexity of pregnancy experiences in women with SLE. Although zero-shot prompting is widely used, it produced lower thematic results in our study, whereas few-shot prompting achieved the most comprehensive and coherent theme extraction. LLM-assisted qualitative analysis may complement human interpretation and has the potential to support the inclusion of non-English language data and diverse patient perspectives in lupus research.
    DOI:  https://doi.org/10.1002/acr.80193
  17. J Med Internet Res. 2026 Oct 20. 28 e96886
       Background: Clinical practice guidelines are central instruments of evidence-based care; yet, in many domains, particularly rare or complex diseases, multiple guidelines coexist and frequently offer conflicting advice. Existing strategies for resolving such divergences are often labor-intensive, opaque, and insufficiently reproducible.
    Objective: This tutorial aimed to develop and demonstrate a reproducible workflow for transforming divergent clinical guideline recommendations and literature-derived white areas into human-reviewed Delphi-ready candidate items.
    Methods: Following a step-by-step tutorial structure, we demonstrate the workflow using penile cancer as a worked example. The process comprises manual selection of clinical guidelines, a systematic PubMed search for literature-derived candidate issues potentially unaddressed by the selected guidelines, selection of 3 complementary large language models (LLMs), and execution of predefined prompts, multimodal recommendation extraction, semantic harmonization, and item-level guideline stance classification. The evaluative process combines a 3-LLM ensemble consistency check with an independent pre-Delphi appraisal by 4 penile cancer experts using a predefined 5-point suitability scale and acceptance threshold.
    Results: In the worked example in the field of penile cancer as a high-stakes use case, the workflow extracted 157 clinical recommendations from 5 guidelines and consolidated them into 68 harmonized guideline-derived items. Stance classification identified 15 uniformly supported items and 53 divergent, ambiguous, or incompletely addressed items. In parallel, literature processing generated 95 harmonized white-area items, of which 9 were classified as true white-area items not addressed by the included guidelines. The final methodological output was a set of 62 human-reviewed Delphi-ready candidate items for subsequent clinical consensus voting.
    Conclusions: This methodology provides a scalable, transparent, and transferable approach for identifying and structuring interguideline inconsistencies and evidence gaps. By combining LLM-assisted analysis with independent external expert appraisal, it supports the reproducible transformation of heterogeneous guideline content into Delphi-ready candidate items.
    Keywords:  Delphi method; artificial intelligence; clinical practice guidelines; consensus methodology; evidence harmonization; large language models; rare diseases; reproducibility; white area analysis
    DOI:  https://doi.org/10.2196/96886
  18. Front Psychol. 2026 ;17 1933405
      Artificial intelligence (AI) is increasingly reshaping qualitative research, quantitative research, and mixed methods research, but practical guidance for conducting, evaluating, and reporting human-AI analytic collaboration remains underdeveloped. Building on AI-augmented transformatizing, a representational-recursive framework that conceptualizes AI as a participant in analytic re-representation rather than merely as a technical tool, this article advances a practice-oriented framework for Human-AI Augmented Mixed Methods Research. The article offers three integrated contributions. First, it proposes the AI-Augmented Transformatizing Protocol, a recursive workflow for preparing representations, for assigning human and AI analytic roles, for generating AI-assisted transformations, for conducting human critique, for refining findings, and for constructing meta-inferences. Second, it introduces quality criteria for assessing methodological rigor in human-AI augmented analysis, including representational fidelity, prompt traceability, recursive transparency, human oversight integrity, algorithmic reflexivity, convergence-divergence interrogation, meta-inferential robustness, and ethical accountability. Third, it presents reporting standards designed to help authors disclose AI systems, model versions, prompting procedures, analytic iterations, researcher interventions, validation/legitimation strategies, ethical safeguards, and limitations. These proposed methodological components are conceptual and practice-oriented rather than empirically validated; accordingly, the worked example illustrates their operational logic, whereas future research is needed to establish their empirical performance, including the validity and reliability of the HAAM-QAM. Rather than treating AI use as an optional efficiency enhancement, the framework positions human-AI collaboration as a structured methodological practice requiring explicit design, documentation, reflexivity, and accountability. By translating the theory of AI-augmented transformatizing into protocols, quality criteria, and reporting standards, this article provides methodologists, researchers, authors, reviewers, editors, graduate educators, mentors, and the like with practical guidance for conducting transparent, rigorous, and ethically responsible AI-augmented mixed methods research in psychology and related fields. In so doing, it extends current reporting guidelines and quality frameworks by addressing the distinctive representational, recursive, and ethical demands introduced when AI participates directly in analysis.
    Keywords:  AI-augmented transformatizing; Artificial intelligence; Human-AI Augmented Mixed Methods Research; Human-AI collaboration; Mixed Methods Research; methodological governance; recursive analysis; reporting standards
    DOI:  https://doi.org/10.3389/fpsyg.2026.1933405
  19. Arch Public Health. 2026 Oct 17. pii: 225. [Epub ahead of print]84(1):
       BACKGROUND: For public health policy research, it is crucial to monitor policy debates to inform evidence-based policymaking. One popular method is to analyze the news for salient frames used to argue for or against a controversial topic. However, this method is time and resource-intensive since domain experts need to manually analyze relevant articles to identify frames. This study presents an approach for using large language models (LLMs) to extract frames and to systematically evaluate results.
    METHODS: Building on the results of a framing analysis of the German media debate about a tax on animal-based foods, we developed a systematic workflow to utilize LLMs to extract all instances of the 10 most frequent manually coded negative frames from 46 news articles. With one prompt per article in a few-shot setting we generated a list of quote/frame pairs. We then evaluated results against human-coded ones and calculated precision, recall, and F1-score.
    RESULTS: GPT-5 achieved 79.8% precision and 79.8% recall, identifying 75.2% of manually coded frames. Extracted quotes generally aligned with human-extracted text sections. Gemini 3.1 Pro Preview yielded higher precision but lower recall, resulting in a comparable F1-score (79.0% vs. 79.8% with GPT-5). These findings suggest the approach is generalizable across different frontier models.
    CONCLUSIONS: This study introduces a workflow for LLM-assisted frame extraction and transparent evaluation against an expert reference standard. Integrating this approach into public health policy coding can save resources and accelerate analysis. Ultimately, this could increase the representativeness and timeliness of data, enhancing our understanding of ongoing public health policy debates.
    Keywords:  Framing analysis; Germany; Large language models; Media analysis; Public health nutrition; Public health policies; Tax on animal-based foods
    DOI:  https://doi.org/10.1186/s13690-026-02089-6
  20. PLoS One. 2026 ;21(10): e0360157
      Large Language Models (LLMs) have evolved from relying primarily on knowledge stored in their parametric memory to using non-parametric memory mechanisms such as Retrieval-Augmented Generation (RAG) to provide and ground knowledge. Recently, LLMs have begun to perform retrieval autonomously within a reasoning loop, an approach referred to as Agentic RAG. This shift introduces new querying behaviours, with implications for the design of retrieval systems that have yet to be evaluated. In this work, we introduce a large-scale dataset (ASQ) capturing how Agentic RAG systems perform multi-step search and reasoning. ASQ contains about 1M agentic search traces collected from open-source Agentic RAG systems over 3 corpora, 5 agent variants, and 6 retrieval settings. Our analysis of ASQ shows that, within this controlled benchmark setting, a single search query can trigger long chains of retrieval calls (up to 186) and follow query reformulation dynamics different from those observed in historical human-search data. Our results also reveal substantial behavioural variation across agent and retrieval configurations: changes in re-ranking and retrieval depth do not consistently affect search workloads in terms of search-call volume and trace length across agents. To our knowledge, publicly available production logs do not expose the agent, dataset, and retrieval configuration associated with each trace. ASQ provides a reusable resource for further analysing agent search behaviour and supporting future information retrieval studies on agentic workloads under controlled settings.
    DOI:  https://doi.org/10.1371/journal.pone.0360157