Artificial Intelligence (AI)-Based simulators versus simulated patients in undergraduate programs: A protocol for a randomized controlled trial

Abstract

Background

Healthcare simulation is critical for medical education, with traditional methods using simulated patients (SPs). Recent advances in artificial intelligence (AI) offer new possibilities with AI-based simulators, introducing limitless opportunities for simulation-based training. This study compares AI-based simulators and SPs in undergraduate medical education, particularly in history-taking skills development.

Methods

A randomized controlled trial will be conducted to identify the effectiveness of delivering a simulation session around history-taking skills to 67 fifth-year medical students in their clinical years of study. Students will be assigned randomly to either an AI-simulator group (intervention) or a simulated patient group (control), both will undergo a history-taking simulation scenario. An Objective Structured Clinical Examination (OSCE) will measure the primary outcomes. In contrast, secondary outcomes including student satisfaction and engagement, will be evaluated following the validated Simulation Effectiveness Tool-Modified (SET-M). The statistical approach engaged in this study will include independent t-tests for group performance comparison and multiple imputations to handle missing data.

Discussion

This study’s findings will provide valuable insights into the comparative advantages of artificial intelligence-based simulators and simulated patients. Results will guide decisions regarding integrating AI-based simulators into healthcare education and training programs. Hybrid models might be considered by institutions in the light of this study, providing diverse and effective simulation experiences to optimize learning outcomes. Furthermore, this work can prepare the ground for future research that addresses the readiness of AI-based simulators to become a core part of healthcare education.

Background

Simulation allows healthcare professionals to acquire and improve their clinical skills in a safe and controlled environment, which is essential for medical education and training. Most traditional simulation modalities rely on simulated patients (SPs) to simulate realistic clinical situations. However, recent advances in computer simulation and artificial intelligence (AI) have led to the creation of AI-based simulators, which offer more advanced features to medical simulation.

For years actors known as simulated patients have played a vital role in medical education. These individuals are trained to portray conditions, symptoms and behaviors adding a human element to simulation exercises. By offering students a space to practice examinations, communication skills and patient interactions, simulated patients enhance the learning experience. Their ability to offer feedback and adjust their responses based on learners’ actions is well known for enhancing the realism of simulations [1, 2].

The use of simulated patients has been shown to improve clinical skills, empathy, and communication among healthcare professionals [3, 4]. However, there are inherent disadvantages to this simulation modality, such as the cost and complexity of researching and training SPs, the potential for variation in SP performance, and the inability to precisely control all simulation delivery elements.

The use of AI-based simulators is a new approach to healthcare simulation. To create highly realistic training situations, these simulators sometimes use virtual reality (VR) and artificial intelligence algorithms. They are extremely accurate when it comes to simulating different physiological reactions and medical situations [5]. Simulators using artificial intelligence have the advantage of being consistent, which allows students to practice repeatedly to develop skills at an affordable cost.

Offering personalized feedback and adaptation, AI-based simulators may also adjust the level of difficulty of simulated scenarios based on the learners’ proficiency [6]. One benefit of this kind of simulator is the ability to facilitate realistic dialogue with patients using natural language processing.

Several studies have shown the benefits of using AI simulators to enhance information retention and boost the expertise of healthcare trainees [7, 8]. Despite all these advantages, AI simulators still face many obstacles related to their high initial cost and implementation, as well as concerns about them introducing the risk of losing the human touch in healthcare training [9, 10].

Methods/Design

Aim

The aim of this randomized controlled trial (RCT) is to determine the effectiveness of AI-based simulators compared to simulated patients delivering a formal simulation training for history taking skills to undergraduate medical students during their clinical years of study.

This study will explore the following questions:

• 1. To what extent do AI-based simulators as opposed to simulated patients (SPs) impact the learning and retention of skills among undergraduate students?

• 2. What are the perspectives and experiences of undergraduate students and instructors when utilizing AI-based simulators versus SPs in healthcare simulation scenarios?

We hypothesize that advanced AI-powered simulators equal or may surpass the outcomes achieved through SP-based simulations, in healthcare training settings.

Design

This study is a randomized controlled trial conducted at Dubai Medical College for Girls, UAE. The CONSORT (Consolidated Standards of Reporting Trials) guidelines will be followed ensuring transparent reporting [11].

Settings and participants

The study will be performed at the simulation center at the Dubai Medical College for Girls. Participants for the study will be fifth-year medical students undertaking their MBBS degree. This program has been replaced by a new MD program starting in September 2023, while the MBBS program has been phased out progressively since then. Students within this program spend the first two years in pre-clinical studies, undertaking general courses with few to no simulation-based training. At the commencement of the third year, students are slowly introduced to the clinical environment, spending more teaching time at a teaching hospital. From years three to five, students are placed in a clinical learning environment with more simulation-based training where history-taking skills are taught repetitively.

Recruitment

Fifth-year medical students will be randomized to either group A where an AI-based simulator will be used to deliver the scenario or to a control group B where a classic simulated patient will be performing the case.

Participants must be a fifth year DMCG MBBS student to be eligible for this trial. Students who are unable physically or mentally to take part in simulation-based training will be excluded. To limit the potential influence of prior exposure, students who participated in similar studies will also be excluded from the recruitment process.

Randomization

Participants will be randomly assigned as shown in Fig. 1 to either group A (experiment) or group B (control) using block randomization method to ensure balanced group sizes. The randomization sequence will be generated using a computer random number generator. To maintain allocation concealment and prevent selection bias, an independent, blinded member from the research team, who is not involved in the study execution, will manage the randomization list and process. Participants will be assigned to their respective groups without the involvement of the investigators to ensure that group allocation remains unknown until the intervention is administered.

Fig. 1

Flow diagram of the proposed RCT

Intervention

Students randomized to the group A (intervention) will receive a simulation session within the simulation center of Dubai Medical College of Girls. The intervention will consist of using a chest pain scenario adapted to ALEX, an AI-based mannikin capable of conducting a humanized dialogue simulating a doctor-patient conversation. With the simulator present in the room like it would be the case of any simulated patient, the entire exchange with the simulator will rely on voice instructions as they would do with any simulated patient. The simulation scenario will consist of 10 min briefing/Orientation to the simulation settings. Students will play then the scenario for approximatively 20 min. A debriefing session will be conducted after the completion of the scenario to discuss the critical points and provide time for reflection. The scenario chosen for this experiment will be tackling a chest pain. The patient will be announced to be just arriving at the emergency department, suffering a chest pain without any companion or family member. Students will have to acquire all details from the patient directly, mobilizing their communication skills for a proper history taking.

Students randomized to the control group will receive the same scenario as the intervention group, but this time the patient will portrayed by a simulated patient. All simulation settings will be same. The simulation scenario will be adapted to an SP modality for case delivery. A simulated patient training will be conducted for the person to understand the scenario and learn the detailed script for the history of the patient in this case.

An observed structured clinical examination (OSCE) will be conducted before and after the delivery of the scenario for both groups to evaluate the history taking skills. After completing the assessment part, students will be asked to fill respond to the Simulation Effectiveness Tool-Modified (SET-M) [12]. This will give an insight about the level of confidence and efficiency of both modalities used.

Blinding

Blinding of investigators and participants is not possible as the subject educators and students will be aware of their allocated arm. Although, the OSCE assessors and data analyst will be kept blinded.

Outcome measures

The primary outcome to be measured in this study will be competency in history taking skills. This will be measured by using a standardized OSCE developed and validated by a subject matter expert. The consistency of this assessment will be ensured across both groups through a standardization training process. This training will include:

• Calibration sessions: examiners will undergo calibration sessions where they will review the OSCE scoring rubric and practice using it on both groups. These sessions will help align all examiners on the evaluation criteria and scoring process.

• Dry run examinations: Examiners will participate in mock OSCEs, with both AI and SP modalities. They will practice applying the evaluation criteria uniformly.

• Continuous standardization: Consistency will be maintained throughout the study via regular debriefing sessions where examiners discuss any potential challenges, they face in applying the scoring criteria. This feedback loop will be instrumental to minimize variability in our assessments.

The effectiveness of the scenario and the students’ perceived competency will also be assessed using the previously validated Simulation Effectiveness Modified Tool (SET-M) [12].

These outcomes will be measured in the control and intervention groups one week after the intervention is implemented.

It is important to mention that a novelty effect related to AI-based simulators might be encountered as a bias in our study, leading to an increased interest or engagement from students for the simple reason of the novelty of the technology. This can potentially affect their performance in the evaluation phase. To mitigate this bias, we plan to introduce both study groups to a familiarization session before getting to the assessment part. This will make students more comfortable with their respective simulation modality and reduce any potential performance enhancement due to novelty. Furthermore, our debriefing sessions will help account for students’ subjective experiences which will differentiate genuine learning outcomes from novelty related excitement.

Analysis

Sample size calculation

We will take a convenient sampling approach, leveraging the accessibility and student readiness within the DMCG settings. This approach is practical and facilitates the recruitment of volunteers by focusing directly on members of the target population.

To determine the sample size, we performed a power analysis seeking a statistical power of 80%, a significance level (alpha) of 0.05, and an expected dropout of 10%. By doing so, the study improves its ability to detect significant differences in learning outcomes and student satisfaction in gamified and conventional simulations.

• Effect Size (d): 0.5 (medium effect size, typical for educational interventions).

• Alpha (α): 0.05 (standard level for Type I error).

• Power (1 - β): 0.80 (standard level for power).

• Total Population (N): 67 students (Year 5 DMCG students).

• Anticipated Dropout Rate: 10%.

Initial Sample Size Estimation for Infinite Population: Without accounting for the small population and dropouts, the original estimate suggested that we would require about 64 participants in total for an independent two-sample t-test with the provided parameters (d = 0.5, α = 0.05, power = 0.80).

Adjusting for Finite Population Size: We are using the Finite population correction (FPC) as the sample size is a significant fraction of the total population. The formula for FPC is:

$$n' = \frac{n}{{1 + \frac{{(n - 1)}}{N}}}$$

Substituting n = 64 and N = 67, the adjusted sample size becomes:

$$n' = \frac{{64}}{{1 + \frac{{63}}{{67}}}} = \frac{{64}}{{1.94}} \approx 33$$

Adjusting for Dropout: Anticipating a 10% dropout rate, we need to inflate the adjusted sample size to ensure enough participants complete the study:

$${\rm{Adjusted}}\,{\rm{Sample}}\,{\rm{Size}}\,{\rm{with}}\,{\rm{Dropout}} = \frac{{33}}{{0.90}} \approx 37$$

Practical Application Considering Total Population: Based on a 10% dropout rate and a total population of 67 fifth-year students, our reasonable projection is that 60 students will participate. But according to our FPC and dropout rates calculations, we require about 37 students to maintain statistical validity. Thus, practically speaking:

Maximally usable students = All available students = 67 (or possibly 60 accounting for dropout).

In this scenario, we plan to use all 67 students (assuming less than planned attrition) or, in a more conservative manner, about 60 students, dividing them as equally as feasible between two groups (30 students each group if the remaining 60 students remain). Our sample size calculation uses this method to optimize the number of participants considering constraints, while focusing on the statistical power required to identify an average effect size. It considers the constraints imposed by the available population and potential dropout rates, thus ensuring that the statistical analysis of the study remains robust and faithful.

Data analysis

The primary outcome which is competency in history taking skills will be analyzed through an independent two-sample t-test to compare the assessment scores between the two groups. While for secondary outcomes like students’ satisfaction resulting from the simulation effectiveness tool-modified (SET-M), we will use t-tests as well for comparison between both groups. In case data is not normally distributed, a non-parametric alternatives like Mann-Whitney U test will be applied.

Missing data will be handled using multiple imputation techniques to ensure the robustness of the analysis. Sensitivity analysis will be conducted to assess the impact of different assumptions about the missing data mechanism by comparing imputed data results with complete case analysis.

The potential interaction effects between group allocation and student characteristics such as prior simulation experience or academic performance will be explored using two-way ANOVA. We will also conduct subgroup analysis to examine if any demographic factors influence the effectiveness of AI-based simulators versus SPs, potentially leading to recommendations for different subgroups of learners. The summary of the results can be organized as per the Table 1 below.

Table 1 Statistical summary of the expected results

Discussion

This study aims to provide insightful information on the use and effectiveness of artificial intelligence simulators compared to simulated patients (SPs) in medical simulation training for undergraduate medical students. Many outcomes can be anticipated like:

• Evidence-based practice: By methodically and rigorously comparing artificial intelligence simulators and simulated patients, several evidence-based elements can be derived. This can potentially influence the field of healthcare simulation as well as program development methods. The study will eventually reveal the potential strengths and weaknesses of each method. As our hypothesis points out, AI-based simulators are expected to excel or at least match the performance of simulated patients. If this proves to be the case, a paradigm shift may be likely in the next few years in healthcare simulation. A more sophisticated and diversified approach to simulation-based education will be discussed in scientific circles to further investigate the optimization of simulation modalities.

• Students’ satisfaction: The study is expected to provide a clear idea of ​​learner satisfaction and perceptions of the simulation experience. These results will help to better understand which method learners prefer and find most engaging for their learning of clinical skills like history taking. It will give an insight to simulation centers and undergraduate medical programs about how they can adapt their simulation technology to accommodate students and enhance their learning experiences.

While we hypothesize that AI-based simulators can match or surpass simulated patients in teaching history taking skills, we must also acknowledge the possibility that our results may show no significance between the two groups or even that AI-based simulators underperformed. In this case, such findings will indicate that the use of AI-based simulators in this context may require more refinement before they can be considered as a replacement to simulated patients in medical education. These findings could stress the importance of human factors and interaction in clinical training as well which AI-based simulators might not be able to compete with currently.

By bridging the gap in this new field of studies, further research can be inspired. As AI is growing fast, such study can lead to more specialized interests in exploring the impact and effectiveness of this new technology on healthcare simulation.

It is important to mention that the relatively small sample size and number of cases limits the generalizability and transferability of the results of this study. This work is an initial pilot for a more detailed exploration into the comparative effectiveness of AI-based simulators and simulated patients. The findings of this study can help inform other larger scale research in the future that might aim at validating these results in a much larger population.