International accreditation of a pediatric hematology/oncology fellowship program in Guatemala: a quantitative assessment

Abstract

Purpose

High-quality graduate medical education programs are essential to expand the care available to children with cancer worldwide. The authors used the Education Program Assessment Tool (EPAT), a standardized tool for evaluating pediatric hematology/oncology (PHO) fellowship programs, to describe the impact of the Accreditation Council for Graduate Medical Education International (ACGME-I) accreditation process.

Methods

The EPAT is a comprehensive, validated tool for evaluating the elements of PHO fellowship programs. The authors used the EPAT in May 2018 and December 2023 to compare the pre-accreditation and post-accreditation status of the Unidad Nacional de Oncología Pediátrica (UNOP) PHO fellowship program.

Results

The authors used the EPAT to identify program gaps and to prioritize interventions to meet ACGME-I accreditation standards. Before accreditation, the overall score of the UNOP PHO fellowship program was 77.2%. The highest score was for Hospital Infrastructure (86.4%) and the lowest score was for Research (63.0%). After accreditation, the overall score was 88.3%. Eight of the 10 EPAT domains had higher post-accreditation scores, with a significant improvement in domain scores overall (P < 0.0001) (Fig. 2). In alignment with the improvement workstreams, the Educational Infrastructure and Evaluation domains had the greatest increases in scores (27% and 25%, respectively). Research continued to have the lowest score (75.9%), but this was improved by 13%.

Conclusions

This study provides quantitative data on the enhancement of a PHO fellowship program in a middle-income country after the process ACGME-I accreditation based on the improvement of EPAT scores. Additional analyses and evaluation tools are needed to identify strategies optimize training approaches and to meet the evolving accreditation standards for health care around the world.

Introduction

Expanding high-quality graduate medical education (GME) programs is essential to increase health care service delivery capacity, especially in the context of marked global health disparities. Internationally, the approach to medical education and GME is shifting toward competency-based training [1,2,3]. Mechanisms that ensure that competent clinicians and agents of change graduate from these programs are crucial to achieve the goal of meeting the health needs of the population [4].

Accreditation is the process by which institutions and training programs are assessed for their compliance with predetermined standards of structure, process, and achievement [4]. Accreditation is therefore central to meeting quality benchmarks. In the United States, the Accreditation Council for Graduate Medical Education (ACGME) evaluates and accredits GME programs [5]. The ACGME International (ACGME-I) aims to improve health care by assessing and improving the quality of physicians’ education through accreditation of GME programs outside the United States [6]. Although accreditation is essential to the oversight of GME programs, its effect on training programs has limited quantitative data [7,8,9]. Internationally, published reports are scarce and have focused on perceptions of value [10, 11].

More than 80% of children with cancer reside in low- and middle-income countries, where health systems frequently lack the resources for comprehensive diagnosis and treatment, resulting in survival rates drastically lower than those in high-income countries [12]. The shortage of pediatric hematologists/oncologists contributes to the limited availability of care for many children with cancer globally [12]. Therefore, the education of professionals through pediatric hematology/oncology (PHO) fellowship programs is crucial for improving access to care.

Because of the need for well-trained pediatric hematologists/oncologists in Central America, the Unidad Nacional de Oncología Pediátrica (National Pediatric Oncology Unit, UNOP) in Guatemala City launched a PHO fellowship program in partnership with the Universidad Francisco Marroquín Medical School and St. Jude Children’s Research Hospital (St. Jude). Given the significance of the UNOP fellowship program for the region and the need for continuous improvements in PHO training, ACGME-I accreditation was sought as a strategy to enhance the quality of the fellowship program and to ensure that it meets the highest standards. In this study, a standardized tool was used to evaluate a PHO fellowship programs to quantify the impact of the ACGME-I accreditation process at UNOP.

Methods

Evaluation strategy

The St. Jude Global Education Program Assessment Tool (EPAT) is a comprehensive, validated tool designed to evaluate the elements of PHO fellowship programs [13]. The EPAT comprises 10 domains: Hospital Infrastructure, Patient Care, Education Infrastructure, Program Basics, Clinical Exposure, Theory, Research, Evaluation, Educational Culture, and Graduate Impact. Each domain includes questions designed to assess the training components necessary to achieve sufficient competencies to care for children with cancer and blood disorders. EPAT questions were derived from published accreditation standards, elements in published conceptual models for graduate medical education standards, and components of evaluation tools developed for other medical specialties. Each question is assigned a value that reflects its importance within the domain. Responses are also scored based on the extent to which each component is addressed. This strategy results in a percentage score for each domain. Descriptive outputs for each domain identify the strengths and areas for improvement of each training program. Consensus for EPAT domains, questions, and scores was completed in a two-stage approached where experts provided input. The first stage was an internal consensus within St. Jude and the second stage was a external panel, identified based on their positions as experts in the field of global pediatric hematology/oncology and in training programs in pediatric hematology/oncology. Furthermore, the EPAT was piloted for validation with 5 training programs in 5 countries, including 2 in Latin America, which represented diverse medical training and patient care contexts. In total, EPAT has 190 questions across the 10 domains. One of the EPAT domain’s questions and scoring is included in Supplement 1. This tool can ultimately help to identify programs strengths and areas of opportunities to help prioritize modifications over time. The EPAT was used as a cross-sectional evaluation in May 2018 and December 2023 to compare the pre-accreditation and post-accreditation status of the UNOP PHO fellowship program. The tool was completed electronically using Qualtrics by the UNOP PHO fellowship program director in both instances.

UNOP pediatric hematology/oncology fellowship program

Guatemala is a middle-income country in Central America with 17.4 million inhabitants and a gross national income per capita of $5,350 US Dollars [14]. UNOP is the only hospital in Central America dedicated exclusively to pediatric cancer. UNOP treats 90% of children with cancer in Guatemala (approximately 600 new patients with pediatric cancer diagnoses per year) in a 82-bed hospital and 2 satellite outpatient clinics.

The UNOP PHO fellowship program started in 2003 and is 3 years in length [15]. The fellows mainly train at UNOP, but also have rotations at local centers to broaden their exposure to pathology, radiation oncology, and flow cytometry. Fellows also rotate at pediatric cancer centers in the United States (St. Jude and Children's Hospital Colorado) and Italy (Hospital San Gerardo in Monza and the Instituto di Tumori in Milan). To address the specialist gap in the region, trainees are selected based on the needs of regional pediatric cancer units for pediatric hematologists/oncologists and their plans for employment after training. To date, 46 physicians from 10 countries in Central or South America have completed the training, and more than 95% of these physicians are currently practicing PHO in the region.

ACGME-I accreditation process

The ACGME-I accreditation process includes 2 primary steps: the accreditation of the institution overseeing a training program and the accreditation of the program itself [16]. For institutional accreditation, the sponsoring institutions are required to support the learning environment by providing proof of commitment to graduate medical education. After the institutional accreditation is obtained, a training program can pursue accreditation. Programs must provide support for an effective learning environment, including theoretical learning and supervised clinical experience with progressive responsibility during training.

Statistical analysis

The improvement in EPAT scores was analyzed using a paired t-test. For this analysis, a p-value of less than 0.05 was considered statistically significant. Statistical analysis was performed with GraphPad Prism, v9.1.

Ethical approval

The institutional review board of St. Jude Children’s Research Hospital approved this study.

Results

Pre-accreditation evaluation

In 2018, UNOP and St. Jude performed an internal comprehensive accreditation readiness evaluation to identify areas requiring modification to meet the ACGME-I standards. As part of this needs assessment, the EPAT was used to identify gaps and to prioritize interventions to enhance the PHO fellowship program. The overall score of the UNOP PHO fellowship was 77.2%. The highest score was for Hospital Infrastructure (86.4%) and the lowest score was for Research (63.0%) (Table 1).

Table 1 Pre-accreditation EPAT domains, areas of focus, and scores

The readiness evaluation identified 4 principal areas of focus that required substantial changes. The required changes were as follows: a) formalizing a fellowship oversight structure with a more clearly defined distribution of roles within the fellowship leadership; b) updating the curriculum and rotation schedule and defining the educational objectives for program components; c) increasing support for fellow research; and d) expanding the systems for fellow, faculty, and program evaluation.

Implementation of priority improvements

A GME committee was established at UNOP with representation from fellows, faculty, and leadership from UNOP and St. Jude, and the GME committee was incorporated into the UNOP institutional organizational structure. Furthermore, the formerly centralized role of fellowship oversight was divided into 3 distinct roles: a fellowship program director, an associate program director, and an institutional education lead (a designated institutional official). The ACGME-I granted accreditation to UNOP as a sponsoring institution on August 25, 2021. The timeline of the accreditation process is shown in Fig. 1.

Fig. 1

Timeline of the ACGME-I accreditation process. Abbreviations: ACGME-I, Accreditation Council for Graduate Medical Education International; EPAT, Education Program Assessment Tool; PHO, pediatric hematology/oncology; UNOP, Unidad Nacional de Oncología Pediátrica

Multiple modifications to the PHO program were required prior to application for ACGME-I accreditation. The didactics schedule and the rotations schedules were updated by introducing dedicated research time and assigned research mentorship. Fellows at UNOP attend didactic lectures by local faculty, but these lecture offerings were expanded and incorporated the St. Jude fellow lecture series. Because no system existed for faculty and program evaluations, a web-based 360-degree evaluation system was created to assess the trainees, the faculty, and the program itself. The fellowship program obtained accreditation on January 22, 2022. With this, UNOP became the first ACGME-I–accredited pediatric hematology/oncology fellowship program in the world. Table 2 describes the changes that were needed across the EPAT domains. These included opening a bone marrow transplant program, which was beyond the scope of the changes prioritized by the accreditation process but was deemed an institutional priority to expand care.

Table 2 Changes implemented for each EPAT domain

Post-accreditation evaluation

In December 2023, approximately 2 years after the initial accreditation, the EPAT was used to re-evaluate the UNOP program and obtained an overall score of 88.3%. Of the 10 EPAT domains, 8 had higher scores after the re-evaluation. There was a significant improvement in domain scores overall (P < 0.0001) (Fig. 2). The greatest score increases were observed for the Educational Infrastructure and Evaluation domains, for which the scores increased by 27% and 25%, respectively. Despite remaining the lowest score (75.9%), the Research domain score improved by 13%.

Fig. 2

Radar plot of the Education Program Assessment Tool score before and after accreditation

Discussion

The mission of accreditation bodies is to improve health care and population health by assessing and enhancing the quality of education for resident and fellow physicians. This study of the impact of the process of ACGME-I accreditation using the EPAT at one institution provides quantitative data on the improvement of a PHO training program in a middle-income country.

Based on the EPAT scores, most of domains had a quantified improvement after the accreditation process. These improvement were consistent with previous reports on the perceived impact of ACGME-I accreditation on fellowship quality [10, 11]. In a report from 8 academic medical centers in 3 countries, nearly 80% of GME leaders and clinicians educators believed that seeking ACGME-I accreditation was worthwhile [10]. The 2 domains that did not improve, Hospital Infrastructure and Graduate Impact, are harder to modify as the former is influenced by the size and scope of the hospital and the latter is a change that would be measurable on a longer timeline than that of this study (5 years), as it would depend on the graduating trainees.

The quantitative evaluation of the value of the process of accreditation is limited by 2 main factors: 1) the lack of standardized, comprehensive tools for quantitatively evaluating GME programs, and 2) the fact that accreditation has been mandatory for many decades in high-income countries, limits the existance of data on pre- and post-accreditation status. Our work has tried to addresses these two factors. First, the EPAT is a systematically-designed quantitative tool designed for PHO fellowship programs. Second, the decision to seek ACGME-I accreditation of the PHO fellowship program at UNOP allowed for the evaluation of a pre- and post-accreditatoin phase.

Although essential and frequently mandatory, GME program accreditation is largely a binary process that focuses on training processes. As a medical community, more systematic and comprehensive systems with which to evaluate training programs and link them to population health needs are vital. Tools such as EPAT can help us build a robust, fit-for-purpose workforce to bridge health gaps in the future [17]. This process is most relevant in low- and middle-income countries, which have large gaps in workforce capacity, but it is also applicable in high-income countries, where the saturation of labor markets affects the integration of GME graduates into the active workforce [18, 19].

Although the UNOP program showed quantifiable improvements based on the EPAT evaluation, areas of opportunity still exist. The long-term impact of accreditation on fellowship quality and graduate competencies remains unknown, but the accreditation process has already improved the educational infrastructure and the fellowship program educational organization, thus providing a better training environment. Given the scarcity of systematically acquired objective data on the impact of fellowship accreditation, data on the UNOP PHO fellowship program will be collected as it develops.

This work has limitations. First, the evaluation relies on a single evaluation tool, EPAT. However, EPAT is an innovative and quantitative tool for evaluating PHO fellowship programs and has been successfully piloted in different training contexts. Our experience in evaluating the UNOP program across timepoints expands upon the validation of EPAT. Second, EPAT was completed by the UNOP fellowship director in both instances and desiarability bias could have influenced the EPAT responses. Nonetheless, since many of the EPAT questions are binary this bias is likely negligible. Third, some of the improvement in the program may be due to changes beyond the accreditation process, including the assessment using EPAT, the areas of opportunity and subsequent enhancement of the program. Nonetheless, these changes would have still be based on a strategic improvement of program processes. Finally, this study focuses on the experience of one institution and one fellowship program. As a pediatric cancer center, UNOP may not be representative of other institutions and GME programs that require their own unique accreditation processes. Additional analyses are planned to explore a broader range of contexts of training and PHO care.

The ultimate goal of medical education systems is to address the health needs of the population [4]; therefore, strong GME programs are essential to health systems. To strengthen the workforce and GME programs, a robust monitoring and evaluation framework based on multidimensional quantitative assessment is needed. Such a framework will enable the education community to optimize training approaches and meet the evolving accreditation standards to improve health care around the world. Our work has tried to add to the existing literature on how quantitative tools can be used to create monitoring systems for GME programs. This approach is not specific to only PHO, but adaptations could be incorporated for most GME programs.