Application of ChatGPT-assisted problem-based learning teaching method in clinical medical education

Abstract

Introduction

Artificial intelligence technology has a wide range of application prospects in the field of medical education. The aim of the study was to measure the effectiveness of ChatGPT-assisted problem-based learning (PBL) teaching for urology medical interns in comparison with traditional teaching.

Methods

A cohort of urology interns was randomly assigned to two groups; one underwent ChatGPT-assisted PBL teaching, while the other received traditional teaching over a period of two weeks. Performance was assessed using theoretical knowledge exams and Mini-Clinical Evaluation Exercises. Students’ acceptance and satisfaction with the AI-assisted method were evaluated through a survey.

Results

The scores of the two groups of students who took exams three days after the course ended were significantly higher than their scores before the course. The scores of the PBL-ChatGPT assisted group were significantly higher than those of the traditional teaching group three days after the course ended. The PBL-ChatGPT group showed statistically significant improvements in medical interviewing skills, clinical judgment and overall clinical competence compared to the traditional teaching group. The students gave highly positive feedback on the PBL-ChatGPT teaching method.

Conclusion

The study suggests that ChatGPT-assisted PBL teaching method can improve the results of theoretical knowledge assessment, and play an important role in improving clinical skills. However, further research is needed to examine the validity and reliability of the information provided by different chat AI systems, and its impact on a larger sample size.

The rapid development of artificial intelligence technology is leading to a profound change in people’s daily lives and work. ChatGPT, as a type of natural language processing technology, can automatically process and generate natural language. The technology has attracted widespread attention and has been referred to as having great significance, and has been widely used in multiple fields. In the field of medical education, ChatGPT has a wide range of application prospects [1,2,3].

In contrast to other professional students, medical students are not only required to complete the basic knowledge of specialized fields but also need to acquire clinical practice capabilities. The problem-based learning (PBL) teaching method places emphasis on student-centered learning and problem-oriented learning, guiding students to explore and solve problems independently [4, 5]. This approach avoids the limitations of traditional lecture-based teaching, where students are not able to fully exert their exploratory and active nature. Previous research has shown that PBL teaching methods can effectively improve the understanding of theoretical knowledge and clinical practice skills of medical students [6,7,8].

However, the combination of PBL and ChatGPT-assisted teaching methods in medical education is still in the exploratory stage [9]. To improve the quality and effectiveness of education, to enhance students’ self-study and team cooperation capabilities, we have introduced the teaching model of PBL and ChatGPT-assisted learning in the field of urology, and conducted a research on medical undergraduates as the research object, through evaluating the results of students’ work, exploring its feasibility and application in medical education.

Materials and methods

Participants

We evaluated the impact of PBL ChatGPT-assisted teaching on medical education of 42 medical interns in Xiangya Medical College of Medicine’s 5th-year medical curriculum. The study population consisted of 23 males and 19 females, who were randomly assigned to either the PBL ChatGPT-assisted (n = 21) or traditional teaching group (n = 21) for a two-week rotation in urology. No participants had previously completed PBL teaching mode. All students participated in a skills training program organized by the same instructor. Randomization was stratified by baseline clinical assessment scores to ensure comparability of the groups.

Study design

This is a prospective, randomized, controlled study. Students were randomly assigned to two groups. The experimental group received PBL and ChatGPT- assisted teaching methods, while the control group received traditional teaching methods. Both groups had the same textbooks, teaching content, instructor, and total teaching time. The study design process flowchart is illustrated in Fig. 1. Ethical approval was obtained from the institutional review board, and informed consent was secured, with special attention to privacy concerns due to the involvement of patient data.

Fig. 1

Study design and flow chart

Instructional implementation

The complexity and individualized treatment needs of renal tumors require students to have a comprehensive understanding of kidney diseases and related clinical skills. In this study, we selected to teach students about the clinical manifestations, history taking, physical examination, diagnostic strategies, differential diagnosis, and treatment options of renal tumors. The curriculum included two parts: theoretical learning and practical skills training. We developed the teaching objectives based on the characteristics of renal tumors and helped students master the relevant content.

PBL ChatGPT-assisted teaching group

Study group formation

The students were divided into four groups, three groups consisting of 5 students and one group consisting of 6 students. Within each group, a communication platform was established, and the group leader was appointed.

Pre-class knowledge transfer

The students were familiarized with the features and potential educational applications of the ChatGPT 4.0 version. Prior to the commencement of the class, the instructor dedicated one session to educating students on the utilization of ChatGPT 4.0 version. Subsequently, the students’ mastery of ChatGPT was assessed through a classroom-based test.

Prior to the class, the teaching objectives were sent to the students. Each group of students communicated with the ChatGPT, followed the teaching objectives, and engaged in the preliminary exploration of relevant content. The students interacted with the artificial intelligence in an interactive way, exploring the dynamic generation of clinical cases based on renal tumors, which included clinical manifestations, history taking, physical examination, diagnostic strategies, differential diagnosis, and treatment options. This provided students with the opportunity to enhance their understanding of various clinical scenarios through the use of the artificial intelligence.

Asking questions

Before the class, the students were given several open-ended questions generated by ChatGPT, which they discussed and explored independently. We have also used ChatGPT to generate simulated patients and conducted simulated interviews with the virtual patients to practice the process of history-taking. This approach enables students to practice their skills in collecting patient histories while engaging in interactive and dynamic learning experiences with ChatGPT. The purpose of this process was to enable students to receive appropriate diagnoses, select appropriate treatments, and apply the knowledge they had learned to analyze and solve clinical problems.

Teaching process

a)Classroom Discussions: Each group of students summarized and reported on problems generated by ChatGPT before, proposing answers in the classroom. Finally, the instructor summarized and reviewed all the problems discussed.

b)Clinical Skills Training: After the discussions, students practiced history taking and physical examination in the patients’ bedsides.

During the entire process, instructor critically reviewed the content provided by ChatGPT and provided instant, personalized feedback to ensure its accuracy. This direct involvement of AI and educators aims to improve learning outcomes by leveraging the scalability of AI and the expertise of educators.

Traditional teaching group

Pre-class knowledge transfer

The instructor designed a typical case of renal tumors and students actively preview relevant knowledge to achieve the learning objectives by studying the case.

Teaching process

The instructor first introduced the clinical cases in detail, explaining the diagnostic reasoning process and emphasizing the importance of history taking and physical examination techniques. We developed multimedia presentations (text, audio, video, images, etc.) that detailed each renal tumor’s clinical manifestations, history taking, physical examination, diagnostic strategies, differential diagnosis, and treatment options in detail. Afterward, students independently conducted patient interviews and physical assessments under the observer’s guidance of the instructor. The teaching time of the traditional teaching group is the same as that of the PBL-ChatGPT group.

Theoretical knowledge exam of two groups

Before the class began, all students took a theoretical exam. Then, the third day after the class ended, they took another exam to evaluate their knowledge of the theoretical material. The maximum score for the exam was 100 points, and a score below 80 was considered a failing grade.

Mini-CEX assessment of two groups

Mini-CEX has been widely accepted as an effective and reliable method for evaluating clinical skills [10, 11]. It includes students’ ability to obtain a patient’s history from the patient and family and to perform physical exams under the guidance of the instructor. The Mini-CEX scoring system uses a nine-point scale and includes seven standards: medical interviewing skills, physical examination skills, humanistic qualities/professionalism, clinical judgment, counseling skills, organization efficiency and overall clinical competence. The scale ranged from unsatisfactory (1–3 points), satisfactory (4–6 points), superior (7–9 points). To maintain consistency, all Mini-CEX evaluations are performed by a single evaluator.

ChatGPT satisfaction survey questionnaire

We evaluated students’ satisfaction and impact of using PBL-ChatGPT on their learning experience. The study used a survey with six questions.(Supplementary Table 1) The survey was completed by students who participated in the PBL ChatGPT-assisted teaching group.

Statistical analysis

Descriptive statistics are represented by the mean ± standard deviation (x ± s) and are compared using independent t-tests. Categorical data is represented by frequency and percentage (n,%) and is analyzed using the chi-squared test when appropriate. All statistical analyses are performed using SPSS (version 23.0, Chicago, IL, USA). A P-value < 0.05 is considered statistically significant.

Results

Theoretical knowledge exam scores

The scores of the two groups of students who took exams three days after the course ended were significantly higher than their scores before the course (PBL-ChatGPT group: p < 0.01;Traditional teaching group: p < 0.01). the scores of the PBL-ChatGPT assisted group were significantly higher than those of the traditional teaching group three days after the course ended (PBL-ChatGPT group: 93.90 ± 3.65;Traditional teaching group: 90.33 ± 4.08; p < 0.01) (Table 1).

Table 1 Distribution of theoretical knowledge exam scores

Mini-CEX evaluation results

All participants completed the Mini-CEX assessment on average within 37 ± 1.3 min and the average time for immediate feedback for each student was 4.3 ± 0.8 min. The PBL-ChatGPT group showed statistically significant improvements in medical interviewing skills, clinical judgment and overall clinical competence compared to the Traditional Teaching group. The Table 2 summarizes the detailed comparison of Mini-CEX scores between the two groups of students.

Table 2 Mini-CEX scores between the two groups of students

Satisfaction survey

The students gave highly positive feedback on the PBL-ChatGPT teaching method. The satisfaction rate was very high, and there were no dissatisfied situations reported. The Table 3 provides a detailed overview of the feedback items and results specific to the PBL-ChatGPT teaching method.

Table 3 Result of ChatGPT satisfaction survey questionnaire

Discussion

The teaching method of PBL encourages students to take an active role in learning. Several studies have confirmed that the appropriate application of PBL teaching methods can overcome the limitations of traditional teaching methods and improve students’ clinical thinking ability [12,13,14]. It enables students to better understand and master theoretical knowledge, and enhance their professional skills [15, 16]. ChatGPT, as a new generation of artificial intelligence language processing models, has brought about a profound change in medical education [17, 18]. It provides extensive medical information, clinical case analysis, and medical skills simulation, helping medical students better understand and master medical knowledge. At the same time, ChatGPT offers more flexible and personalized learning ways, catering to the diversified learning needs of students [19,20,21]. By combining PBL teaching methods with ChatGPT, we can leverage their advantages and enhance the quality of teaching and the training of medical professionals.

Our research shows that the use of ChatGPT as a teaching aid in the PBL method can improve the results of theoretical knowledge assessment, and play an important role in improving clinical skills. The high level of satisfaction among students towards the use of ChatGPT indicates that artificial intelligence has the ability to increase students’ engagement. This positive response can be attributed to the personalized and interactive nature of the AI experience, which meets the diverse learning styles of students. However, some students have also reported that the AI system sometimes struggles to understand the specific language context, leading to misunderstandings and inaccurate answers. Students also noticed that ChatGPT has limitations in understanding technical terms in surgery. In the process of teaching, we have observed that the probability of errors occurring in AI is approximately 15%. Some of these errors can be resolved by adding qualifying words. However, another portion of the errors arises from AI’s inability to filter out accurate answers, necessitating teachers to review the answers provided by AI during the instructional process.

ChatGPT can personalize education content based on individual performance indicators, thereby promoting more precise and effective learning experiences [22,23,24]. The advantages of PBL-ChatGPT in theory learning and clinical skills prove that artificial intelligence technology can assist in improving students’ practical operation skills and clinical thinking ability. ChatGPT can provide extensive medical information, but its answers depend on pre-trained data [25]. If ChatGPT’s training data has biases and is incomplete, the algorithm may have biases, leading to information incomplete, outdated or inaccurate. Therefore, before PBL teaching begins, it is crucial for students to acquire the accurate way to communicate with ChatGPT. It is essential for students to develop skills in constructing high-quality prompts, including proficiently selecting keywords, so as to effectively convey tasks or problems to the artificial intelligence model. As students often fail to create effective prompts, this can lead to answers with biases. Therefore, during PBL teaching, teachers should closely review the accuracy of information provided by ChatGPT. Medical education is not just a process of transmitting knowledge, but also emphasizes cultivating students’ interpersonal communication and communication skills, as well as emphasizing cultivating students’ emotional intelligence and empathy ability. Although ChatGPT has the ability to process natural language, it cannot provide the ability of emotional communication and understanding. When medical students rely too heavily on ChatGPT, they will lose trust in human relationships, and the relationship between people and people will be limited to “human- machine” level [26]. Over-reliance on ChatGPT can lead to the lack of interconnectedness of information, resulting in students’ thinking becoming fixed, lacking innovative thinking, and further affecting the effectiveness of medical education [27]. Therefore, ChatGPT should be viewed as a supplement to teaching methods, rather than a substitute. During PBL teaching, the advantages of ChatGPT should be utilized as a teaching aid, while teachers should pay attention to providing appropriate guidance and supervision to ensure that students receive reliable relevant information, while maintaining the personalized learning experience of individualization and interaction.

This research has some limitations. Firstly, the sample size was small, which may limit the generalizability of the results. Additionally, the survey used in this study may have limitations in terms of accuracy and validity. Self-reported data may be prone to bias, and the design of the survey may not have effectively captured the experiences and feelings of the participants. The current study did not analyze the individual effects of PBL and ChatGPT-assisted learning on the outcomes. Future research is needed to separately compare the results of these two learning methods in order to determine the impact of ChatGPT on learning effectiveness. Moreover, further research is needed to assess the effectiveness of other chat AI systems in enhancing PBL education outcomes, and to explore the unique features and limitations of each system, in order to determine their potential advantages and disadvantages in improving the PBL learning experience.