JMIR AI. 2026 Oct 28. 5
e103273
BACKGROUND: Clinical practice guidelines (CPGs) translate research evidence into actionable recommendations, but their development faces mounting challenges. With dozens of trials and systematic reviews published daily, the traditional 18- to 24-month development cycle struggles to incorporate emerging evidence, and manual screening, synthesis, and quality assessment consume thousands of person-hours per guideline. The resulting knowledge translation pipeline is slow, inconsistent, and quickly outdated.
OBJECTIVE: This review aimed to chart the computational methods empirically tested for automating CPG development, the tasks they address, and their evaluation measures, and to map where that evidence is concentrated and where it is absent for living guidelines.
METHODS: The review followed the Joanna Briggs Institute scoping review framework and PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) guidance. We searched PubMed and Scopus for studies published between January 2001 and December 2025, supplemented by backward and forward citation searching. Eligible studies used automated methods for a guideline development task, reported original research with empirical testing, and gave sufficient methodological detail. Two reviewers independently screened records and extracted data. Findings were charted across 4 pillars: retrieval, extraction, encoding, and evaluation.
RESULTS: A total of 47 studies met eligibility criteria, spanning 2001 to 2025, with a marked increase after 2015 and a surge in large language model (LLM) work in 2024-2025. Pillar counts (nonexclusive) were as follows: retrieval, 6; extraction, 15; encoding, 26; and evaluation, 17. In retrieval pillar, query expansion achieved 80% recall, while semantic search tools traded recall for specificity (98%). In extraction, rule-based natural language processing achieved an F-measure of 81.96% for structured clinical actions and 71.75% for patient states, and hybrid ontology methods covered 86% of eligibility criteria; LLMs performed well on structured fields but degraded on interpretive content. In encoding, approaches progressed from formalisms such as Guideline Interchange Format version 3 (GLIF3) and guideline elements model toward Fast Healthcare Interoperability Resources (FHIR)-based representation; semiautomated tools assisted guideline encoding, and logic verification detected inconsistencies and coverage gaps missed by human reviewers. In evaluation, LLM appraisers assigned higher Appraisal of Guidelines for Research and Evaluation II (AGREE II) scores than human appraisers, by a mean difference of 12.5% in one study (95% limits of agreement -30.6% to 55.5%; intraclass correlation coefficient 0.753) and by 22 to 60 percentage points across domains in another.
CONCLUSIONS: Automation shows promise for bounded, well-structured guideline development tasks, while robust evidence for end-to-end, production-scale living guideline pipelines remains limited. No included study demonstrated a validated pipeline connecting all 4 pillars in an operating guideline program, and evidence on clinical safety, workflow integration, and cost-effectiveness is largely absent. Realizing living guidelines will require interoperability standards such as FHIR, and evaluation in real guideline production rather than retrospective benchmarks.
Keywords: AGREE II; Appraisal of Guidelines for Research and Evaluation II; GRADE; Grading of Recommendations Assessment, Development and Evaluation; artificial intelligence; automation; clinical practice guidelines; computer-interpretable guidelines; decision support systems; evidence-based medicine; living guideline; machine learning; natural language processing