Skip to main content

Table 3 Mean, standard deviation values, and Inter-Rater reliability level of evaluators

From: Is AI the future of evaluation in medical education?? AI vs. human evaluation in objective structured clinical examination

Square Knot Criteria

Inter-rater

Reliability

Evaluator

H1

H2

RT

AI1

AI2

α

κ

M(Sd)

M(Sd)

M(Sd)

M(Sd)

M(Sd)

While holding both ends of the string in the palms of both hands, used the thumb of the non-dominant hand to pull the string in the other hand to create a cross.

0.170

0.093

1.14

(0.9)

1.28

(0.7)

1.60

(0.7)

2

(0.0)

1.74

(0.4)

Positioned the string between the thumb and index finger of the other hand after crossing.

0.088

0.110

0.98

(1.0)

1.05

(0.9)

1.55

(0.9)

1.66

(0.5)

1.67

(0.5)

Passed the string through the crossed section.

0.119

0.087

1.62

(0.8)

1.17

(0.9)

1.53

(0.8)

1.98

(0.1)

1.64

(0.6)

Adjusted the string’s ends to a neat position and tied a square knot.

0.169

0.047

1.69

(0.6)

1.07

(0.9)

1.66

(0.6)

1.98

(0.1)

1.67

(0.5)

For the reverse knot, took the string from the thumb side of the cross, creating a new cross.

0.174

0.043

0.79

(0.9)

0.72

(0.9)

1.26

(0.8)

1.60

(0.5)

1.64

(0.5)

Brought the thumb and index fingers together and rolled the cross inside the hands to the other side.

0.248

0.066

0.84

(0.9)

0.79

(0.9)

1.26

(0.8)

1.97

(0.2)

1.67

(0.5)

Squeezed the other string between the thumb and index fingers, passing it back through the cross to the front side.

0.212

0.081

1.14

(0.9)

0.76

(0.9)

1.22

(0.9)

1.93

(0.3)

1.66

(0.5)

Straightened the ends of the string and tightened the knot.

0.259

0.033

1.60

(0.7)

0.71

(0.9)

1.41

(0.8)

1.98

(0.1)

1.59

(0.5)

Checked the tightness of the knot.

0.350

0.002

0.74

(0.9)

0.78

(0.8)

0.95

(0.9)

2

(0.0)

1.76

(0.5)

  1. H1: Human Evaluating from Video 1, H2: Human Evaluating from Video 2, RT: Real Time Human Evaluator, AI1: ChatGPT, AI2: Gemini Flash, M: Mean, Sd: Standard Deviation, α: Krippendorff’s, κ: Fleiss