Skip to main content

Table 7 Mean, standard deviation values, and Inter-Rater reliability level of evaluators

From: Is AI the future of evaluation in medical education?? AI vs. human evaluation in objective structured clinical examination

Urinary Catheter Placement Criteria

Inter-rater

Reliability

Evaluator

H1

H2

RT

AI1

AI2

α

κ

M(Sd)

M(Sd)

M(Sd)

M(Sd)

M(Sd)

Introduced themselves to the patient.

0.139

-0.028

1.79

(0.6)

2

(0.0)

1.98

(0.1)

1.65

(0.6)

2

(0.0)

Explained the procedure to be performed and obtained consent for pelvic examination (informed consent obtained).

0.156

-0.041

1.48

(0.7)

1.98

(0.1)

2

(0.0)

1.81

(0.4)

1.81

(0.5)

Washed and dried their hands (verbal explanation was sufficient).

0.073

-0.001

1.48

(0.9)

1.92

(0.3)

2

(0.0)

1.85

(0.4)

1.90

(0.4)

Wore sterile gloves (saying it was sufficient).

0.102

-0.011

1.60

(0.7)

1.96

(0.2)

2

(0.0)

1.92

(0.3)

1.90

(0.4)

Indicated the injection site visually (upper outer quadrant).

0.297

-0.108

1.35

(0.5)

2

(0.0)

1.98

(0.1)

1.73

(0.5)

1.85

(0.4)

Cleaned the penis three times with gauze soaked in antiseptic solution, starting from the external urethral orifice in expanding circular motions.

0.103

-0.019

1.94

(0.3)

2

(0.0)

2

(0.0)

1.85

(0.4)

1.54

(0.8)

Applied lubricant gel to the tip of the chosen Foley catheter.

0.026

-0.018

1.92

(0.4)

1.96

(0.2)

1.92

(0.4)

1.85

(0.4)

1.79

(0.4)

With one hand, directed the tip of the catheter while holding the penis with the other hand, gently advancing the catheter tip into the urethra.

0.311

-0.032

0.83

(0.9)

1.98

(0.1)

2

(0.0)

1.85

(0.4)

1.71

(0.5)

Held the syringe with the active hand as if holding a pen.

0.146

-0.034

1.27

(0.9)

2

(0.0)

1.85

(0.4)

1.88

(0.3)

1.96

(0.2)

When the catheter tip reached the perineal level, lowered the patient’s penis and aligned it parallel to the body’s long axis.

0.018

-0.009

1.75

(0.7)

1.96

(0.2)

1.92

(0.4)

1.92

(0.3)

1.71

(0.6)

Continued to advance the catheter upward, parallel to the body’s long axis.

0.124

-0.041

1.33

(0.9)

1.98

(0.1)

1.90

(0.4)

1.85

(0.4)

1.46

(0.7)

Observed the urine flow from the expected tip of the catheter / noted the urine output.

-0.002

0.061

1.85

(0.5)

1.96

(0.2)

1.94

(0.3)

1.85

(0.4)

1.85

(0.5)

After urine flow started, advanced the catheter 2–3 cm further.

0.122

-0.010

1.83

(0.6)

1.98

(0.1)

2

(0.0)

1.79

(0.4)

1.48

(0.8)

Attached the urine bag (it could have been attached earlier).

0.206

-0.043

1.13

(1.0)

1.98

(0.1)

1.88

(0.3)

1.92

(0.3)

1.40

(0.7)

Administered saline to the balloon’s pathway using a syringe

0.339

-0.055

0.69

(0.9)

1.96

(0.2)

1.85

(0.4)

1.94

(0.3)

1.33

(0.8)

Slowly pulled the catheter back and, after feeling the balloon rest on the bladder neck, advanced it 1–2 cm.

0.139

-0.028

1.79

(0.6)

2

(0.0)

1.98

(0.1)

1.65

(0.6)

2

(0.0)

Suspended the urine bag below the bladder, ensuring proper positioning

0.156

-0.041

1.48

(0.7)

1.98

(0.1)

2

(0.0)

1.81

(0.4)

1.81

(0.5)

  1. H1: Human Evaluating from Video 1, H2: Human Evaluating from Video 2, RT: Real Time Human Evaluator, AI1: ChatGPT, AI2: Gemini Flash, M: Mean, Sd: Standard Deviation, α: Krippendorff’s, κ: Fleiss