Skip to main content

Table 1 Mean, standard deviation values, and Inter-Rater reliability level of evaluators

From: Is AI the future of evaluation in medical education?? AI vs. human evaluation in objective structured clinical examination

IM Injection Criteria

Inter-rater

Reliability

Evaluator

H1

H2

RT

AI1

AI2

α

κ

M(Sd)

M(Sd)

M(Sd)

M(Sd)

M(Sd)

Introduced themselves and explained the procedure to the patient.

0.001

0.196

1.86

(0.5)

1.88

(0.4)

1.91

(0.4)

2

(0.0)

1.95

(0.3)

Checked the accuracy and suitability of the medication and verified it was the correct patient (verbal confirmation is sufficient).

0.117

0.269

1.30

(0.9)

1.51

(0.9)

1.26

(0.9)

1.95

(0.2)

1.91

(0.4)

Ensured patient privacy (verbal confirmation is sufficient).

0.437

0.076

0.47

(0.9)

0.60

(0.8)

0.79

(0.9)

1.91

(0.3)

1.86

(0.5)

Syringe Preparation (Checked the expiration date of the syringe and ensured the packaging was intact), stated that they prepared the medication by drawing it into the syringe.

0.125

0.054

1.05

(0.4)

1.12

(0.9)

1.21

(0.5)

1.63

(0.5)

1.56

(0.6)

Indicated the injection site visually (upper outer quadrant).

0.043

0.072

1.70

(0.6)

1.81

(0.5)

1.84

(0.5)

1.98

(0.2)

1.95

(0.2)

Cleaned the injection area with an antiseptic cotton swab, wiping outward in a circular motion from the center, covering a radius of approximately 5 cm.*

0.049

-0.007

1.81

(0.4)

1.98

(0.2)

1.91

(0.3)

2

(0.0)

1.67

(0.7)

Placed a dry cotton swab between the 4th and 5th fingers of the assisting hand.

0.002

0.065

1.77

(0.6)

1.79

(0.6)

1.88

(0.3)

1.88

(0.3)

1.72

(0.5)

Stretched the skin at the injection site using the thumb and index finger of the assisting hand.

-0.005

0.177

1.77

(0.7)

1.79

(0.6)

1.84

(0.5)

1.86

(0.4)

1.98

(0.2)

Held the syringe with the active hand as if holding a pen.

0.001

0.289

1.88

(0.5)

1.93

(0.3)

1.88

(0.4)

2

(0.0)

1.98

(0.2)

Inserted the needle perpendicular to the skin.

-0.007

0.071

1.91

(0.4)

1.93

(0.3)

1.91

(0.4)

1.98

(0.2)

1.88

(0.3)

Pulled back the syringe plunger slightly with the assisting hand to check for blood.

-0.005

0.077

1.81

(0.5)

1.84

(0.4)

1.95

(0.2)

1.86

(0.4)

1.72

(0.7)

Injected the medication into the muscle by pressing the syringe plunger with the assisting hand (no actual medication will be injected into the model)

0.033

0.184

1.74

(0.5)

1.77

(0.5)

1.88

(0.3)

2

(0.0)

1.65

(0.8)

Removed the needle at the same angle and speed as insertion while applying light pressure to the injection site with a cotton swab using the assisting hand.

0.005

0.030

1.98

(0.2)

1.98

(0.2)

1.95

(0.2)

2

(0.0)

1.88

(0.4)

Disposed of the syringe and other waste materials in appropriate disposal containers (without recapping the needle).

0.058

0.056

1.60

(0.7)

1.86

(0.4)

1.91

(0.4)

1.98

(0.2)

1.72

(0.6)

Informed the patient that the procedure was complete and/or said, “Get well soon.”

0.044

0.330

1.67

(0.8)

1.77

(0.6)

1.74

(0.7)

2

(0.0)

2

(0.0)

  1. H1: Human Evaluating from Video 1, H2: Human Evaluating from Video 2, RT: Real Time Human Evaluator, AI1: ChatGPT, AI2: Gemini Flash, M: Mean, Sd: Standard Deviation, α: Krippendorff’s, κ: Fleiss