Development of a Core outcome set and outcome measures for Artificial Intelligence-based conveRsational agents in hEalthcare: CARE study

Artificial intelligence-based conversational agents, such as chatbots and virtual health coaches, are increasingly being integrated into healthcare for interventions like mental health support, chronic disease management, and health education.However, despite their rapid adoption, significant challenges remain in evaluating their effectiveness, safety, and real-world impact. Existing studies report a wide heterogeneity of outcomes, making it difficult to compare results, synthesize evidence, or establish best practices. Currently, there are no defined core set of outcomes and no standardised outcome measures for artificial intelligence-based conversational agent to use in healthcare. Without standardized outcome measures, healthcare providers, policymakers and developers lack a unified framework to assess whether these conversational agents truly improve care quality, reduce disparities, or justify their cost.
Thus, a core set of outcomes must be defined for artificial intelligence-based conversational agents in healthcare.This study seeks to address these gaps by developing the first COS and standardized outcome measures specifically for AI-based conversational agents in healthcare.

Aim
To develop a core outcome set and a set of outcome measures for artificial intelligence-based conversational agents in healthcare. The objectives are aligned to the COMET Handbook 4-steps to develop a core outcome set.


Objectives
1. To identify outcomes reported in studies evaluating artificial intelligence-based conversational agents in healthcare. (WP1: Systematic review)
2. To define a core outcome set for studies evaluating artificial intelligence-based conversational agents in healthcare. (WP2: Delphi study and stakeholder consensus meeting)
3. To identify outcome measures of the core outcome set and define the validity and reliability of these measures. (WP3: Systematic review)
4. To reach consensus on the final list of core outcomes and outcome measures for studies evaluating artificial intelligence-based conversational agents in healthcare. (WP4: Stakeholders meeting with Nominal Group Technique)


Impact:
This project will identify and establish a core outcome set with recommended outcome measures for studies evaluating artificial intelligence-based conversational agents in healthcare. A core outcome set, and a set of recommended outcome measures can increase research engagement of all stakeholders and strengthen the evidence base of related outcomes in AI-based conversational agent interventions.

Contributors

YUXIA ZHANG,Director,Department of Nursing, Zhongshan Hospital Fudan University.
YAMIN YAN,Research nurse,Zhongshan Hospital Fudan University.
YICHEN KANG,Research nurse,Zhongshan Hospital Fudan University.

Further Study Information

Current Stage: Ongoing
Date: May 2025 - May 2027
Funding source(s): To be confirmed


Health Area

Disease Category: Other

Disease Name: N/A

Target Population

Age Range: 16 - 90

Sex: Either

Nature of Intervention: Other

Stakeholders Involved

- Clinical experts
- Consumers (caregivers)
- Consumers (patients)
- Patient/ support group representatives
- Policy makers
- Researchers

Study Type

- COS for clinical trials or clinical research
- COS for practice

Method(s)

- Consensus meeting
- Delphi process
- Nominal group technique (NGT)
- Systematic review

1. To identify outcomes reported in studies evaluating artificial intelligence-based conversational agents in healthcare. (WP 1: Systematic review)
2. To define a core outcome set for studies evaluating artificial intelligence-based conversational agents in healthcare. (WP2: Delphi study and stakeholder consensus meeting)
3. To identify outcome measures of the core outcome set and define the validity and reliability of these measures. (WP3: Systematic review)
4. To reach consensus on the final list of core outcomes and outcome measures for studies evaluating artificial intelligence-based conversational agents in healthcare. (WP4: Stakeholders meeting with Nominal Group Technique)

Linked Studies

    No related studies


Related Links

    No related links