Assessment of anaesthesia trainees using performance metrics derived from electronic health records: a longitudinal cohort analysis

Abstract

Background

The development of competencies in anaesthesia residents is assessed by direct observation of their performance and written and oral examinations. Little is known about how residents’ competencies are reflected by objective data in anaesthetic records. We investigated whether performance metrics derived from electronic anaesthesia records are associated with anaesthesia residents’ progress of training and European written examination timing and results.

Methods

We recruited 46 anaesthesia providers who trained at the Vienna University Hospital between September 2013 and June 2021, and we reviewed the anaesthesia records of all cases they managed during the study period. We derived six performance measures based on perioperative data routinely collected: intraoperative hypotension and hypothermia, glycaemic control, postoperative nausea and vomiting prevention, lung-protective ventilation, and postoperative kidney injury. We evaluated the association between these quality metrics with training level and written exam completion status in anaesthesia residents after adjusting for patient and surgical factors.

Results

We found a statistically significant association between the level of training and most performance measures. The probability of preventing intraoperative hypotension increased (OR 1.16, 95% CI 1.12 – 1.20) with the level of training, as did the probability of preventing hypothermia (OR 1.08, 95% CI 1.05 – 1.11) and administering appropriate postoperative nausea and vomiting prophylaxis (OR 1.21, 95% CI 1.15 – 1.27). However, the odds of preventing acute kidney injury decreased (OR 0.91, 95% CI 0.83 – 0.99), as did the use of lung-protective ventilation (OR 0.94, 95% 0.92 – 0.97). All participating residents who took the written exam passed it, precluding a direct pass versus fail comparison, but the exam completion status was associated with increased odds of lung-protective ventilation (OR 1.42, 95% CI 1.03 – 1.95) and decreased odds of preventing intraoperative hypotension (OR 0.7, 95% CI 0.49 – 0.99). Glycaemic control was not associated with either of the training milestones.

Conclusions

The training level of anaesthesia residents had a significant association with several performance metrics. Passing the written exam only had a modest effect. Performance measures based on patient outcomes and intraoperative care may serve as assessment methods for anaesthesia residents’ progress tracking.

Background

Assessment of medical residents is an essential part of specialty training programs. Formative assessment provides information to deliver corrective and constructive feedback to residents to guide them towards achieving their educational goals. Summative assessment at important educational milestones allows high-stake decision-making about graduation from a program [1].

The assessment process typically involves a combination of multiple methods, partly due to the limited empirical evidence supporting each method. Direct observation, written tests, peer reviews, (self-)assessment in simulated environments or direct patient-physician interactions to assess the competence level acquired by medical trainees [2]. As each method has strengths and limitations, combining multiple methods should provide a more comprehensive picture of medical trainees’ competence. Written examinations are appropriate tools for assessing theoretical knowledge and can even predict actual clinical performance [3]. While assessment and certification processes are very diverse within Europe, the European Diploma in Anaesthesiology and Intensive Care (EDAIC) examination, a two-part supranational examination covering basic sciences and clinical subjects remains the most important assessment for many European anaesthesia residents [4]. In Austria, although taking the EDAIC Part I examination is mandatory, its timing is self-chosen.

Ideally, the assessment of medical trainees should cover technical and non-technical skills, knowledge, and attitudes. Additionally, it should provide insights into daily practice and habits, over mere snapshots from disparate days, cases, or isolated practical skills. An important insight into daily clinical practice might come from measurable patient outcomes related to interventions during anaesthesia care [5]. Residents becoming competent clinician should be able to prevent adverse outcomes with appropriate interventions during the perioperative period. For example, intraoperative hypotension is a modifiable risk factor for postoperative complications, including postoperative nausea and vomiting (PONV), stroke, myocardial injury, acute kidney injury (AKI), and death [6, 7]. Hypothermia can cause significant postoperative morbidity and mortality, particularly coagulopathy and infections [8]. PONV are among the most common causes of patient dissatisfaction and might affect long-term outcomes [9, 10]. Lung-protective ventilation can prevent postoperative respiratory complications [11]. Finally, both hyperglycaemia and hypoglycaemia have been associated with poor outcomes in surgical patients [12].

Electronic health records (EHR) document anaesthesia care and record data related to the patient outcomes [13]. These data provide valuable insights into perioperative management. Such data management systems could provide a different approach to assessing residents' competence in managing patients during anaesthesia care beyond the rather simple observation of skills or answering multiple-choice questionnaires.

We aimed to investigate whether performance metrics derived from anaesthetic records could assess the clinical performance in anaesthesia residents. Specifically, the performance metrics derived from EHR were the ability to avoid hemodynamic instability and hypothermia, appropriate use of PONV prophylaxis, prevention of AKI, judicious use of invasive ventilation, and adequate glycaemic control.

Methods

Study design and objectives

The aim of this retrospective, single-centre study was to investigate the use of routinely collected perioperative data from anaesthesia records to assess the performance of anaesthesia trainees. The specific objectives were to analyse if there was an association between anaesthesia residents’ clinical performance and their training level and between the self-chosen timing and result obtained at the European standardised written exam and the performance measures. The study was conducted at the Department of Anaesthesia, Intensive Care and Pain Medicine at the Medical University of Vienna and the Vienna University Hospital after obtaining approval from the Ethics Committee of the Medical University of Vienna (Ethik-Kommission der Medizinischen Universität Wien, Chairperson Martin Brunner) and its Data Protection Committee (Number 1694/2021 on December 23, 2021). Informed consent was required from all the trainees included in the study. However, the Ethics Committee waived the requirement for informed consent from the patients included.

Participants and data collection

We recruited anaesthesiologists who were in training or had completed their specialty training at the Vienna University Hospital. Participation was voluntary, and we obtained written informed consent from each participating physician. Participants were recruited between January – June 2022. We collected the start and the end dates of the anaesthesia specialty training, and the participation date and the result obtained at the written EDAIC examination, if available, during an interview by the principal investigator.

Procedure date and the provider ID identified cases managed by the study participants during their training. We included patients who underwent procedures requiring general anaesthesia, regional anaesthesia, or procedural sedation between September 1, 2013, and June 1, 2021. We excluded residents who were in training at our institution for less than six months and, for the residents who were included in the study, we only analysed cases managed after the first two months of training—consisting mostly of enrolment and orientation—had passed. We also excluded patients undergoing cardiac surgery or non-operative obstetric procedures, managed by more than one provider (e.g., handed over to another anaesthesiologist), organ procurement from deceased donors, patients with American Society of Anaesthesiologists (ASA) physical status of 5 or 6, and cases requiring only local anaesthesia or anaesthesia provider in stand-by.

We collected patient characteristics, such as sex, weight, height, ASA physical status, functional capacity, and comorbidities from anaesthesia records. Case-related data included the date and time of surgery, the type of surgical intervention (minor versus major, elective versus urgent), the type of anaesthesia, the resident managing the case, vital signs (e.g., blood pressure and body temperature), blood glucose, serum creatinine levels, monitoring parameters (e.g., tidal volume), and medications used (glucose, insulin, and antiemetic drugs).

Performance measures

To assess residents’ performance, we used a set of six performance measures calculated specifically for this study, that were modelled after the quality improvement measures developed by the Multicenter Perioperative Outcomes Group [14]. We computed each performance measures as a binary outcome variable using clinical data queried from the anaesthesia records database on a per-patient basis. We defined AKI as an increase in serum creatinine more than 1.5 times within seven days postoperatively compared to the baseline or an increase by greater than or equal to 0.3 mg/dL within 48 h postoperatively.

Intraoperative hypotension occurred if intraoperative records revealed sustained low blood pressure (mean arterial blood pressure < 65 mmHg for longer than 15 min).

Appropriate glycaemic control was considered an adequate response to hyperglycaemia (i.e., glucose levels > 200 mg/dl) or hypoglycaemia (i.e., glucose level < 60 mg/dl), namely administration of insulin or glucose, respectively, or recheck within 90 min after the abnormal value was documented. We defined lung-protective ventilation as the use of set (as opposed to spontaneous or manual) tidal volumes lower than 8 ml/kg predicted body weight during general anaesthesia.

Hypothermia was defined as at least one documented body temperature < 36° within 30 min before or 15 min after the end of anaesthesia. Appropriate postoperative nausea and vomiting (PONV) prophylaxis involved administration of two different classes of antiemetic drugs in patients with at least one risk factor for PONV.

Data analysis

Due to the exploratory nature of this work, we did not perform a formal sample size calculation; instead, we used all available data between September 1, 2013, and June 1, 2021. We calculated the time in training as the time elapsed since the start of anaesthesia residency. Exam completion status was represented as binary indicator with two assigned values: zero for cases managed before and one after the passing the written exam.

Categorical variables are described as counts and percentages, continuous variables as medians and quartiles due to asymmetric distributions.

To investigate the association between the outcomes constituting the performance measures and the characteristics of anaesthesia providers, Generalised Linear Mixed Models were used at the patient level. Thus, the model reflects the way the data were recorded, since each performance measure was available for each patient. Each model included a random resident ID to account for an association between cases managed by the same resident. For each binary outcome constituting a performance measure, a model was fitted with the model’s dependent variable as the respective binary performance outcome. The statistical analysis plan included the following independent variables: the time in training and two binary variables for the results obtained at the written exam (pass or fail) and the written exam completion status (before and after participating in the exam). However, as all participants who took the exam also passed it, we excluded exam results from the analysis. Initial models assessed each independent variable of interest separately. Given the potential for interaction, we subsequently fitted a model for each outcome including both independent variables of interest and their interaction term to assess whether the effect of the time in training on the binary outcomes was modified by the exam completion status. In each model, we included patient and anaesthetic factors as potential confounders for the effect on the performance measure outcomes: patient age, sex, ASA physical status, functional status, cardiac comorbidities (arterial hypertension, coronary heart disease, myocardial infarction, heart failure New York Heart Association Class (NYHA) I, heart failure NYHA II, heart failure NYHA III, heart failure NYHA IV), respiratory comorbidities (chronic obstructive pulmonary disease (COPD), emphysema), BMI, diabetes and its treatment, type of surgery (minor versus major, planned versus urgent), time of the surgery (daytime versus nighttime), and the type of anaesthetic (general, neuraxial, regional, sedation, or combinations thereof). The urgency of the procedure was documented by the anaesthesia provider, whereas we classified procedures into major or minor based on a Delphi Consensus published by the European Surgical Association [15].

We imputed missing binary performance indicators as worst and best case to provide sensitivity analyses in addition to the primary analysis using only non-missing performance measures. We used multiple imputations for missing adjustment variables assumed as 'missing at random'.

We used SAS 9.4 (SAS Institute Inc., 2016) for all statistical analyses. We considered two-sided p-values below 0.05 statistically significant. We did not perform any adjustment for testing multiple performance outcomes since each of the six outcomes covers a different aspect of residents’ performance.

Results

Forty-six anaesthesiologists participated in this study, of whom10 participating physicians completed their training during the study period; 26 were still in training as of June 1, 2021, while another 10 started their residency before September 1, 2013. In total, 28,350 cases managed during the study period by the participating trainees met the inclusion criteria. Tables 1 and 2 summarise participants’ characteristics. Of all the cases included, 4051 cases were performed before the responsible providers took the exam, which all participants passed.

Table 1 Characteristics of the anaesthesia providers included in the study

Table 2 Patient characteristics^a

The percentage of missing binary outcomes varied between 0 for hypothermia and glycaemic control and 56.5% for AKI. The breakdown of missing data and the overall frequencies for all performance indicators are presented in Table 3.

Table 3 Performance indicators

The association between the time in training and the performance

The time in training, defined as the time elapsed between the start of the specialty training and the date of the case, was significantly associated with all performance measures except glycaemic control after adjusting for patient and surgical factors (Figs. 1 and 2). We found that the odds of preventing intraoperative hypotension increased with the trainee’s experience. With every doubling of the time spent in training, the odds of preventing intraoperative hypotension were 15.6% higher (OR 1.16, 95% CI 1.12 – 1.19, p < 0.001). The odds for appropriate PONV prophylaxis administration were 21% higher with every doubling of the time in training (OR 1.20, 95% CI 1.15 – 1.27, p < 0.001). A similar result, although with a smaller effect, was found for prevention of intraoperative hypothermia (OR 1.08, 95% CI 1.05 – 1.11, p < 0.001). The odds for preventing AKI were lower in more experienced trainees (OR 0.91, 95% CI 10.83 – 0.99, p = 0.028), and the odds for lung-protective ventilation decreased with the time spent in training (OR 0.94, 95% 0.92 – 0.97, p < 0.001). We found no statistically significant association between the time in training and the odds for appropriate management of glycaemia (OR 1.05, p = 0.381).

Fig. 1

Association Between Time in Training and Performance Measures

Legend: Odds Ratios (OR) for the six performance metrics based on the level of training, estimated from the generalised mixed linear models. Each point represents the odds ratio for one of the performance metrics, with 95% confidence intervals (CIs) shown as error bars. The dashed reference line indicates an odds ratio of 1

Fig. 2

Estimated Least Squares Means Across Time in Training: Adjusted Probability of the Performance Metrics

Legend: Estimated probability (Least Squares Means) for the six performance metrics across the range of the time in training by ASA physical status. Each panel represents a different performance metric: (a) prevention of intraoperative hypotension; (b) prevention of acute kidney injury; (c) intraoperative normothermia; (d) appropriate PONV prophylaxis; (e) adequate glycemic control; (f) lung-protective ventilation

The association between successful participation in the written examination and the performance measures

Exam completion status was significantly associated with two of the performance measures after adjusting for patient and surgical factors: the odds for lung-protective ventilation were 42% higher after the exam (OR 1.42, 95% CI 1.03 – 1.95, p = 0.031), but the odds of preventing intraoperative hypotension were lower (OR 0.70, 95% 0.49 – 0.99, p = 0.041). We did not find a statistically significant association between the exam completion status and the rest of the performance measures.

Sensitivity analysis

Sensitivity analyses, which replaced missing performance indicators by best- and worst-case result, confirmed the association between the time in training and the performance indicators. The odds of preventing intraoperative hypotension were higher in more experienced trainees after imputing the missing performance measure as worst- (OR 1.16, 95% CI 1.12 – 1.19, p < 0.001) and best-case (OR 1.13, 95% CI 1.10– 1.17, p < 0.001). Sensitivity analyses yielded similar results for appropriate PONV prophylaxis and lung-protective ventilation. Conversely, in the case of AKI, sensitivity analyses produced diverging results, with lower odds of preventing AKI in more experienced trainees in the worst-case analysis (OR 0.91, 95% 0.84 – 0.98, p = 0.013) and higher odds of preventing AKI in the best-case analysis (OR 1.03, 95% CI 1.01 – 1.05, p = 0.015). The odds of preventing intraoperative hypotension were lower after taking the written examination after imputing the missing binary performance measure as worst- (OR 0.70, 95% CI 0.50 – 0.99, p = 0.44) and best-case (OR 0.69, 95% CI 0.50 – 0.97, p = 0.032). Sensitivity analyses for all other performance measures did not yield statistically significant results.

Interaction effects between time in training and exam completion status

We found no statistically significant interaction between time in training and exam completion status for three of the six binary outcomes in the confounder-adjusted model: prevention of AKI, appropriate management of glycaemia, and prevention of hypothermia. For these outcomes, the effect of the time in training on the odds of the outcomes can be considered consistent across the exam completion status and were similar to the results from the separate models reported above, not statistically significant except for prevention of intraoperative hypothermia (Supplemental Table 1).

For prevention of intraoperative hypotension, adequate PONV prophylaxis, and lung-protective ventilation, a significant interaction was found between exam completion status and the time in training (each p < 0.001 for the interaction term). In cases managed after passing the written exam, each doubling of the time in training was associated with 18.7% higher odds of preventing intraoperative hypotension (OR = 1.19, 95% CI 1.15–1.23). Similarly, the odds of adequate PONV prophylaxis increased with the time in training (OR 1.26, 95% CI 1.19—1.34). Lung-protective ventilation, on the other hand, was less likely to be employed by more experienced trainees after passing the written exam (OR 0.92, 95%CI 0.90—0.94). In contrast, the confidence intervals for the effect of the time in training before passing the written exam indicated non-significant effects (Supplemental Table 2).

Discussion

This retrospective study investigated whether the time in training and passing the standardised anaesthesia specialty exam influenced the residents' performance as reflected by six metrics derived from perioperative data. We hypothesised that data routinely collected in anaesthesia records reflects competence gradually acquired during anaesthesia training.

We found that the time spent in training had a significant effect on the performance of anaesthesia residents across several – but not all – performance measures, as it was associated with favourable outcomes: the odds for intraoperative hypotension and intraoperative hypothermia were lower in patients managed by trainees with more extensive anaesthesia experience. More experienced anaesthesia trainees were also more likely to use both appropriate PONV prophylaxis and, surprisingly, excessively high tidal volumes. Successful participation in the standardised written anaesthesia exam was significantly associated with two performance metrics. The odds for intraoperative hypotension were higher after the written exam; however, use of lung-protective ventilation was also more common after successfully taking the exam. The findings of the models that included the interaction term between time in training and exam completion status were consistent (higher odds of preventing intraoperative hypotension and hypothermia, appropriate PONV prophylaxis, but also lower odds of preventing AKI and employing lung-protective ventilation with more time in training), suggesting that our conclusions on the association between the experience and the performance metrics remain robust regardless of whether interaction is explicitly accounted for.

The use of electronic anaesthesia records to assess providers' skills has been proposed before [16]. In anaesthesia, for example, the time required for anaesthesia induction summarised and reported on a per-provider basis, as an automatic tool to monitor efficiency, is available in some data management software. However, the broad evaluation of clinician skills based on performance metrics derived from patient outcomes and clinical data is not a well-established method yet. In a study on 70 anaesthesia residents, Sessler et al. found that intraoperative blood pressure management was not associated with either exam score or clinical competence evaluation but concluded that evaluations based on electronic health records could supplement existing evaluation methods [17]. In a similar study on nurse anaesthetists, the amount of hypotension was not a useful performance indicator, as the difference between different providers was not significant [18]. Conversely, our study shows that in a longitudinal analysis over a more extended period and with a larger sample size, the progress made by anaesthesia trainees can be tracked using similar performance indicators derived from anaesthesia records.

While we were unable to analyse differences between passing and failing candidates as all participants passed the exam, our findings suggest that successful participation in the written examination at the self-selected time had a more limited effect on the performance metrics. Interestingly, exam completion status and the time in training had opposite effects on two of the performance metrics. The odds for intraoperative hypotension were lower as the residents approached the end of their training, but intraoperative hypotension was more likely to occur after the residents passed the exam. Residents were also less likely to employ lung-protective ventilation during their more senior years, whereas the written examination had the opposite effect. We can only speculate that avoiding intraoperative hypotension indicates the provider's skill, while lung-protective ventilation requires rather theoretical knowledge.

The indicators we chose, derived from a previously described set of quality metrics, were selected based on data availability to assess the trainees' competence [14]. If competence is defined as "the habitual and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values, and reflection in daily practice for the benefit of the individuals and communities being served”, these metrics are not enough by themselves to assess 360 degrees of residents’ competence [19].

We must acknowledge several limitations of our study due to its retrospective design. First, the impact of missing data could be significant. Although we considered explanatory variables missing "at random" and we imputed them appropriately, we cannot exclude informative missingness for the occurrence of AKI. We relied on postoperative serum creatinine levels to define AKI, which are measured mostly in patients at risk or with signs of AKI, such as oliguria, and the sensitivity analyses for worst- and best-case provided conflicting results. Second, some of the data we collected has been manually entered by the providers, so not all performance indicators have the same accuracy. Automatically collected data, such as intraoperative blood pressure and tidal volumes, are generally more reliable than data charted manually, such as comorbid conditions or drugs administered. However, technical issues, particularly when no data is manually entered in the anaesthesia records, can lead to a significant proportion of cases missing the outcome data. Third, we did not perform any comparisons between the providers, as the Ethics Committee and Data Protection Committee did not allow it. Fourth, we selected the performance metrics based on data availability in the anaesthesia records, but the choice and the definition could be further refined, since changes in binary outcome definitions can lead to substantial differences in the proportion of cases that met the outcome definition. For example, we considered only hypotensive episodes longer than 15 min, which occurred in 10.5% of cases, because we chose to focus on the trainees’ response to hypotension and consider the clinical significance of its duration. For glycaemic control, the broad range of acceptable values that did not require an intervention (60 mg/dl – 200 mg/dl) and the high percentage of minor procedures in healthy patients lead to only 0.9% of cases that met our outcome definition of glycaemic control. However, in retrospect, a more fitting definition of lung-protective ventilation would be quantitative rather than binary or include a similar temporal criterion. This is particularly important because, in certain cases, such as laparoscopic procedures, patients may require hyperventilation, and tidal volumes can unexpectedly rise above the intended values due to factors like increased chest wall compliance. Fifth, although we adjusted for several patient and surgical factors, residual confounding cannot be excluded and its impact on patient outcomes and some of the performance metrics we analysed can be significant. For instance, the higher odds of AKI in more experienced trainees could indicate that more senior residents are assigned to more difficult cases and the model, even after adjusting the model for several variables, did not fully account for the case mix. Finally, we were only able to recruit trainees who passed the written examination, which could suggest a selection bias and precluded one of the initial study objectives. Future research could aim to include a broader range of participants with varying levels of preparedness to ensure a more distributed range of exam outcomes.

Despite these limitations, our study suggests that performance indicators derived from anaesthesia records could serve as a new assessment tool that complements other established methods. As anaesthesia medical education is moving from time-based training to competency-based training, evaluation based on electronic health records might be beneficial, as it allows progress tracking and benchmarking between trainees and institutions and can always provide feedback continuously to the trainees and consultant-level anaesthetists alike in the form of a dashboard or regularly as personal reports on specific performance metrics. Unlike workplace-based daily assessments, which are often perceived as a burden, automatic assessment based on electronic health records could provide another view on residents’ performance rather than a snapshot [20]. Although impersonal, this type of assessment is less subjective. Furthermore, if “the assessment of competence […] should provide insight into actual performance ([…] when not observed)” [21], automatically derived indicators might be the appropriate tool to make visible what usually is not observed. Furthermore, this type of continuous assessment is more likely to accurately reflect the tacit knowledge and experience compared to a written examination.

However, several factors can limit the utility and adoption of automated assessment tools derived from electronic health records. Since computing performance indicators requires large amounts of data, this method is only suitable for environments where highly granular clinical data is routinely collected, and digital records are available. Now, this applies mostly to anaesthesia, intensive care medicine, and emergency medicine, but with the spread of digitalisation other specialties are likely to follow [22]. Furthermore, teachers' and learners' attitudes towards data-driven assessment can also delay its adoption and the relatively small number of physicians we were able to recruit might indicate medical professionals’ reluctance towards this type of assessment. It is, therefore, essential to recognise that assessment of medical trainees using performance measures derived from digital medical records cannot replace traditional assessment methods with in-person feedback and is at best a complementary assessment tool.

Although it is generally accepted that the widespread use of electronic patient records in the clinical practice significantly impacts medical education, assessment in medical education has remained largely unaffected [23]. Future research should address their validity and robustness, comparing EHR-derived methods with established assessment methods, but also investigate the attitudes of teachers and learners towards their implementation, identifying barriers to adoption, and ways to optimise their use to advance medical education and professions. Comparing trainees and consultants using the same performance metrics would further consolidate the generalizability of our findings by clarifying skill progression and retention and the impact of experience on patient outcomes.

Conclusions

This single centre, retrospective study demonstrated an association between time in training and clinical performance as reflected by performance metrics derived from electronic anaesthesia records. While these metrics may have potential as complementary assessment tools to track training progress and development of competencies in anaesthesia residents, further research is needed to validate their reliability and utility.