Skip to content

P4: Technology for Clinician Integrated Care

Context

Clinicians are presented with a wealth of information about patients from different sources -- for example, pathology systems, medical imaging and EHRs -- and consequently, retrieval of patient data relevant to a consultation or management plan is challenging and risks missing important "signal" in a voluminous and noisy combinations of sources. This is especially problematic in time-limited consultations. Information needs also vary between different professionals and clinical context, which means a single model of information retrieval even for a single kind of data (for example, free-text keyword searching in EHRs) is inappropriate. A large-language-model (LLM)-based system which can parse important information from the electronic health record (EHR) for clinicians in a range of contexts, which can be queried by clinicians using natural language and verified by patients, was proposed at an industry sandpit held in Glasgow as part of the (Phase One development work)[./development-stage.md].

Aim

In this driver project we will develop and validate a non-knowledge based clinical decision support system for clinical staff caring for people with MLTCs in a range of contexts. The system is intended to surface pertinent patient information to aid clinicians, and will share this information with patients so that they can also verify its accuracy. Our phase 1 work emphasised the frustration people with MLTCs experienced on having to repeat clinical histories to multiple professionals, and those with lower health literacy (often associated with lower income and disability) who are less able to comprehend and articulate clinical complaints are more likely to use healthcare services.

Rapid scoping review

A recent (November 2024) systematic review demonstrates that LLMs in healthcare is a rapid growth area of research, with five published and 22 ongoing trials included. Little is known, however, about the potential for LLMs to parse useful information for clinicians caring for people with MLTCs. We will address this gap by conducting a rapid scoping review of current and ongoing research in this area, following the Joanna Briggs Institute methodology.

Review aims

The aim of this review will be to:

  • map features of pre-existing interrogable LLM solutions providing summarised clinical information, and
  • identify potential implementation barriers to these solutions.

Data sources

We will search six databases (MEDLINE, Embase, CINAHL, Scopus, ClinicalTrials.gov and the International Clinical Trials Registry Platform) from 2018 to present.

Search strategy

Searches will employ a range of terms related to the technology including "large language model", "LLM", "generative pre-trained transformer", "GPT", "bidirectional encoder representations from transformers", and "BERT". We will engage information specialists to develop and refine the search strategy.

Eligibility criteria

Any published peer-reviewed academic study and pre-registered study report which describes an LLM being used to parse clinical information will be eligible for inclusion.

Screening

A minimum of two authors will be required to independently screen each title and abstract, with disagreements resolved by a third reviewer. The same process will be applied to full-text screening.

Data extraction and analysis

Standardised data extraction forms will be used, capturing the following information: author, year of publication, country, healthcare context (e.g. primary/secondary care), study design, technology type, notable features, any quantitative or qualitative impact of the technology on clinical practice and patient outcomes, and barriers to successful implementation described by the authors. We will summarise the extracted data in a conceptual map of technologies and features, to represent what is currently understood about LLMs for clinical note parsing. We will synthesise described implementation barriers and explore whether particular associations exist between barrier type and technology type, features, or context.

Co-design process

Drawing on the findings of the scoping review, we will work with multiple stakeholders to co-design and evaluate a clinical record parsing LLM to aid clinicians in accessing relevant information across several clinical contexts. This process will be streamlined by using pre-existing foundation models, thereby limiting the amount of computational resource and training data required.

Participants and sampling

All participants and partners will be purposively sampled to ensure a range of perspectives are incorporated. Clinical specialties will include (at a minimum) representatives from:

  • emergency care
  • primary and out-of-hours care
  • outpatient clinics
  • community-based services (e.g. district/ community psychiatric nursing).

Industry/engineering contacts with expertise in this area have been identified during SysteMatic phase 1. We will recruit people with MLTCs from testbed areas. Clinical stakeholders/experts and patients/patient experts may need to be resampled for development/evaluation if they do not represent the user profiles developed during codesign. Some drop-off is expected with multi-phase co-design work, therefore we will plan for rolling recruitment as the phases progress.

Methods

Development and validation of the tool will involve participatory user-centred methods across four stages: design, develop/integrate, evaluate, and iterate.

Phase 1: Design

We will hold codesign workshops with representatives from a range of clinical disciplines, engineering, industry, eHealth, and patients, to understand the specific needs of different stakeholders. We will also use these workshops to identify suitable clinical MLTC profiles for initial validation studies, focusing on clinical settings where succinct and interrogable patient summaries will be of use in caring for people with MLTCs (e.g. emergency department triage nurses, GPs/primary care practitioners, outpatient clinics).

To make this driver project tractable (given its necessarily open-ended scope) in this Design phase, we will also identify two discrete priority clinical settings (and their MLTC profles) to take forward for development and integration.

Phase 2: Develop/integrate

We will iteratively develop and train a prototype system (based on an open-source foundational model) which can be tailored to these profiles. Using repeated Delphi methods, a panel of patients and clinical experts representing the identified profiles will prioritise features suggested by industry partners and eHealth throughout development, with the aim of arriving at consensus.

The result will be two prototype clinician-facing tools (CFTs) to produce abstractively summarised patient information for examplar use-case patients with the MSPM (musckulo-skeletal, pain and mental illness) (MLTC phenotype)[../people-insight/persona-set.md].

Seamless workflow integration and interoperability (the lack of which is a common implementation barrier) will undoubtedly be a priority. Colleagues in eHealth will work closely with industry partners to ensure compatibility with current clinical systems (e.g. Trakcare, primary care systems).

Phase 3: Evaluate

The prototype will be validated against ground-truthing by clinical experts in the relevant disciplines, who will identify the most important features in each case. These will be compared with the features surfaced by the model. We will create exemplar summaries generated by the model alongside exemplar summaries generated by participating clinicians directly from "raw" EHRs. To ensure clinician 'ground truth' summaries capture relevant information (aligned with patient's goals and motivations for MLTC care), the clinician summaries will be presented during participatory workshops with patients to understand whether they find them relevant to their experiences and priorities from living with MLTC.

  • Interventions:

  • Two prototype CFTs (one for each clinical setting) for abstractive summarisation of from at least one EHR data source.

  • Comparator:

  • Clinician summarisation task (in each clinical setting) without assistance from prototype CFT

  • Outcome Measures:

  • Primary: Acceptability and usability of CFTs using the technology acceptance model (TAM).
  • Secondary: Qualitative comparison (by consensus) of the quality and relevance of clinician-derived vs CFT produced summaries across the two clinical settings and each MLTC use case.

  • Process evaluation: Semi-structured interviews with healthcare professionals to explore experiences, barriers, and facilitators. 

  • Analysis:

  • Quantitative: Descriptive statistics where relevant (e.g. from structured measurements capturing outputs from the application of TAM for the CFTs)
  • Qualitative: Thematic analysis of participant interviews.

Phase 4: Iterate

We will retain the flexibility to iterate the design and develop/integrate stages if the output of the evaluation stage does not meet stakeholder requirements. Endpoints will be quantitative measures of accuracy and qualitative evaluation of patient/clinician feedback.