Randomized controlled trial evaluating the benefit of the app-based clinical decision support system for the management of venous thromboembolism

Abstract

Introduction

Anticoagulant prescription and management are crucial steps of management of venous thromboembolism (VTE). The Clinical decision support systems (CDSS) are designed and developed to aid healthcare professionals in making informed decisions.

Objectives

To develop and assess the efficacy the mobile app-based CDSS for the management of VTE.

Methods

A randomized controlled trial was conducted, enrolling participants aged ≥ 20 years who were one of the following categories: medical student, intern, internal medicine resident, or hematology fellow. Participants were randomly assigned to either app-based CDSS (DECIDE-COAG^®) or control groups. Participants were tasked with prescribing or planning the management of 15 clinical vignettes covering four domains: anticoagulant dosing, perioperative management, management of anticoagulant associated bleeding and diagnosis of VTE. The primary outcome was the mean percentage of accuracy score.

Results

From September 2023 through November 2023, a total of 126 participants were enrolled. The mean percentage of accuracy score was significantly higher (95.6%) in participants using app-based CDSS compared with those in control group (41.3%), a mean difference of 54.5%, 95% CI 49.8–59.2, p < 0.001. The satisfaction mean score was 4.57 ± 0.49 for accessibility, 4.73 ± 0.45 for app stability, 4.94 ± 0.25 for user interface, 4.78 ± 0.42 for simplicity, 4.98 ± 0.13 for assistance in decision-making and 4.90 ± 0.30 for overall satisfaction.

Conclusions

App-based CDSS (DECIDE-COAG^®) demonstrates a significant improvement in the accuracy of anticoagulant prescriptions and VTE management. Further studies that specifically investigate the clinical benefits of app-based CDSS in real-world clinical practice are warranted.

Clinical trial number

TCTR20241117004.

Introduction

Venous thromboembolism (VTE) comprises two main conditions: deep vein thrombosis (DVT) and pulmonary embolism (PE). The mainstay treatment and prophylaxis of VTE is anticoagulant. While the anticoagulant is effective for the treatment of VTE, bleeding is an inevitable complication. In the large observational study, the incident rate of bleeding in patients prescribed warfarin was 3.5% per 100 person-year [1]. Direct oral anticoagulant (DOAC) demonstrated an association with approximately 30% lower risk of major bleeding compared to warfarin [2]. Patients encountering bleeding related to anticoagulant are at risk of hospitalization and mortality [1, 3].

Therefore, prescribing proper anticoagulant, considering factors including type, dosage, route of administration and dose adjustment according to patients’ characteristics is crucial. Patients receiving a non-recommended dose of anticoagulant were found to be associated with an increased risk of death [4]. Anticoagulant prescription error occurred 8.3-40.3% of medical errors reports [5, 6]. It was reported that the early-career physicians faced difficulties in making decisions about prescription of anticoagulant and had significantly more errors compared with attending physicians [6].

The Clinical decision support systems (CDSS) are designed and developed by using an evidence-based medicine approach to assist healthcare professionals in making informed decisions regarding individual patients’ treatment [7]. This system allows healthcare professionals to integrate specific clinical data relevant to the patient’s condition and provide specific recommendations. Previous studies demonstrates that the use of CDSS may improve VTE care and anticoagulant management [8, 9].

Therefore, this current study aimed to develop and assess the efficacy of a mobile app-based CDSS for medical professionals for the management of VTE.

Materials and methods

Phase I App-based CDSS development

A CDSS was developed for diagnosis and anticoagulant management for VTE or other indications. Important domains, for example, VTE diagnosis, anticoagulant dosing and dose adjusting, anticoagulant management during peri-operation or bleeding were listed. The system utilizes a computer-generated algorithm according to patients’ clinical variables, for example, body weight, age, serum creatinine or creatinine clearance, current anticoagulant use, type of procedures, etc. The algorithm’s recommendation adhered to international guidelines [10,11,12,13]. Subsequently, a mobile app interface was developed specifically for iOS and Android systems. We decided to integrate the CDSS into a mobile application rather than embedding it in an electronic health record system, as we believe the mobile application would have greater impact and generalizability for users, including physicians, trainees, medical students, and other medical personnel who are not tied to a specific health record system.

App-based CDSS validation

The mobile app underwent evaluation for validity and usability by experts, comprising hematologists, respirologists, critical care specialists, cardiologists and pharmacists. The evaluation covered 4 domains: accuracy of the content, reliability, up-to-date recommendations and practicality in clinical practice (Supplementary 1). Experts rated each domain in three categories —agree, partly agree and disagree. Additional comments for each domain assessment were collected, leading to revision in the final version of the mobile app.

Phase II Evaluation of the efficacy of the mobile app-based CDSS

We conducted a prospective randomized trial to evaluate the efficacy of app-based CDSS. The trial was conducted at a university-based hospital in Thailand from September 2023 – November 2023. The intervention (app-based CDSS) was compared with usual practice (control). This trial was approved by the research ethic board of Faculty of Medicine, Chiang Mai University (number 028/2022).

Participants

Participants aged 20 years or older were eligible for enrollment if they were trainees in one of the following categories: medical student (4^th, 5^th, 6^th clinic), intern, internal medicine resident, or hematology fellow. Those who refused to participate or had previously tested or used the app-based CDSS were excluded. Informed consent was obtained from all participants prior to their involvement in the trial.

Randomization and allocation

Participants were randomized in 1:1 ratio into app-based CDSS and control groups based on a computer-generated randomization number. The randomization was stratified by the level of education (medical students, interns, internal medicine residents and hematology fellows). The assigned groups were printed on paper and retained in the opaque-sealed envelopes. This trial was conducted in accordance with the CONSORT statement.

Trial procedure

After obtaining consent, participants were allocated to either the app-based CDSS or control groups. Those in app-based CDSS group received tablets containing the app-based CDSS and were provided the orientation to its functions, Participants in the control group were instructed to prepare online or offline materials relevant to the VTE treatment.

Participants in both groups were presented with 15 clinical vignettes covering four topics: anticoagulant dosing, perioperative anticoagulant management, management of anticoagulant associated bleeding and diagnosis of VTE. Throughout the trial, participants in the app-based CDSS group were permitted to use the app in aid in making informed decisions. In the in app-based CDSS group, only DECIDE-COAG^® application was allowed. Meanwhile, those in the control group had the option to use both online resources (e.g. Uptodate^®, Google, Pubmed) or offline materials, excluding the app-based CDSS. Each participant received scores based on the accuracy of the prescription and treatment plan for each clinical vignette. The answer sheets of each participant were collected and scored by the investigators. Each question had only one correct answer. The clinical vignettes, clinical questions, and answers were reviewed by the investigators in collaboration with experts from various specialties, as mentioned above. The outcome assessors were blinded to the participants’ allocated group.

Satisfaction assessment

Participants in the app-based CDSS group were requested to complete a questionnaire assessing their satisfaction after using the app. The questionnaire included 5 domains: including accessibility, stability of the app during use, user interface, simplicity and clarity, assistance in decision-making and overall satisfaction (Supplementary 2). Participants rated each domain on a scale 0 to 5, where 0 indicated lowest satisfaction and 5 indicated highest satisfaction.

Statistical analysis and Sample size calculation

The primary outcome was the mean percentage of accuracy. It is calculated from the score that answered correctly in the exam. The total full score is 15 points from a total of 15 clinical vignettes comparing between the app-based CDSS and control groups. The sample size was determined through the power analysis. We hypothesized that participants using app-based CDSS group would achieve a 20% higher score compared to the controls. The estimated sample size of 126 (63 for each group) was calculated to provide the trial with 80% power at a two-sided alpha level of 0.05.

Continuous variables were presented as mean with standard deviation or median with range, as appropriate. Categorical variables were expressed as numbers and percentages. The difference in mean accuracy was assessed using a Student’s t-test. Means difference with 95% confidence interval (CI) were reported. A pre-specified subgroup analysis was conducted based on the level of education and domain of clinical vignettes. Trend analysis was applied to test the difference between education groups. A p-value less than 0.05 was considered statistically significant.

Results

Phase I app-based CDSS development

A CDSS for VTE management was developed across five domains, covering VTE diagnosis, warfarin dosing recommendations and INR-based dose adjustment, DOAC dosing recommendations, perioperative anticoagulant management, and anticoagulant-associated bleeding management. The app was submitted to the App Store and Playstore under the name DECIDE-COAG^®. Examples of screenshots are demonstrated in Fig. 1 The app-based CDSS was evaluated by 12 experts across all domains (Fig. 2).

Fig. 1

Examples of screenshots from the app-based CDSS (DECIDE-COAG^®)

Fig. 2

Evaluation of validity of the app-based CDSS by experts

Experts expressed the following agreement percentages: for warfarin dosing, accuracy 82%, credibility 100%, modernization 100%, and clinical practicality 91%; for DOAC dosing recommendations, accuracy 100%, credibility 91%, modernization 100%, and clinical practicality 91%; for perioperative management, accuracy 100%, credibility 100%, modernization 100%, and clinical practicality 91%; for the management of anticoagulant-associated bleeding, accuracy 82%, credibility 100%, modernization 100%, and clinical practicality 91%; and for the diagnosis of VTE, accuracy 64%, credibility 73%, modernization 82%, and clinical practicality 64%. In the evaluation of all topics, there was no topic on which the experts disagreed.

Phase II Evaluation of the efficacy of the mobile app-based CDSS

From September 2023 through November 2023, a total of 126 participants were enrolled and underwent randomization with 63 assigned to app-based CDSS group and 63 assigned to control group. Table 1 demonstrated baseline characteristics of participants. The characteristics of participants were similar in two groups. The mean age was 25.33 ± 3.01 years, 49.21% of participants were male, 33.33% were medical students, 20.63% were interns, 39.68% were internal medicine residents and 6.35% were hematology fellows. Among participants, 64.29% reported having experience using other applications to aid clinical decision-making.

Table 1 Characteristics of participants

Primary outcome

The mean accuracy was significantly higher (95.6%) in participants using app-based CDSS compared with those in control group (41.3%), a mean difference of 54.5%, 95% CI 49.8–59.2, p < 0.001 (Fig. 3). Subgroup analysis revealed that there were significant difference of means accuracy between participants in app-based CDSS group compared to controls across all subgroup medical students with a mean difference of 69.5%, 95% CI 64.2–74.9, p < 0.001, interns with a mean difference of 51.3%, 95% CI 45.5–57.1, p < 0.001, internal medicine residents with a mean difference of 47.2%, 95% CI 41.8–52.6, p < 0.001 and hematology fellow with a mean difference of 31.7%, 95% CI 16.2–47.1, p = 0.002, Fig. 4. Trend analysis revealed significant difference in the scores between education groups in both app-based CDSS group (P = 0.0002) and controls (P < 0.0001). Subgroup analysis, categorized by the domain of VTE management, revealed that participant allocated to app-based CDSS group achieved significantly higher score as compared to those in control group across all domains (Fig. 5). The mean differences were as follows: for anticoagulant dosing 43.5%, 95% CI 36.9–50.1, p < 0.001, for perioperative management 67.5%, 95% CI 61.7–73.2, p < 0.001, for bleeding management 47.6%, 95% CI 34.3–60.9, p < 0.0001, and for VTE diagnosis 12.9%, 95% CI 4.3–21.1, p = 0.003.

Fig. 3

Mean percentage of accuracy comparing between participants in app-based CDSS (purple bar) versus control groups (yellow bar)

Fig. 4

Mean percentage of accuracy comparing between participants in app-based CDSS (purple bar) versus control groups (yellow bar), subgroup analysis according to level of education

Fig. 5

Mean percentage of the accuracy comparing between participants in app-based CDSS (purple bar) versus control groups (yellow bar), subgroup analysis according to domain of VTE management

Satisfaction assessment

Satisfaction was evaluated in 63 participants in app-based CDSS group. The mean score was 4.57 ± 0.49 for accessibility, 4.73 ± 0.45 for app stability, 4.94 ± 0.25 for user interface, 4.78 ± 0.42 for simplicity, 4.98 ± 0.13 for assistance in decision-making and 4.90 ± 0.30 for overall satisfaction, Fig. 6.

Fig. 6

Mean of satisfaction score in participants who were using app-based CDSS

Discussion

This current study demonstrates the process of mobile app-based CDSS development for management of VTE. The mobile app “DECIDE-COAG^®” was found to be valid and beneficial in facilitating informed decision making for healthcare professionals.

During the development process, we observed that there was less expert consensus on the diagnosis of VTE because it was a score in the first round of evaluation before improving the application. The evaluation for validity and usability by experts, comprising hematologists, respirologists, critical care specialists, cardiologists and pharmacists. Some experts noted that D-dimer levels tend to increase in pregnant patients, particularly during the third trimester. We decided to adopt the ASH guideline because it specifically mentions the D-dimer cutoff in pregnant patients, making the recommendation clearer for users [13]. We refined the algorithm for VTE diagnosis after adapting it based on the experts’ comments.

A previous study by Sexena et al. assessed the effectiveness of using CDSS to assist and warn against prescription errors in five hospitals in the United States. Following the implementation of the CDSS, physicians’ prescription behavior changed by 42%. More importantly, a warning system reduced the incidence of prescriptions error [14].

This current study demonstrated that participants who utilized the app-based CDSS achieved significantly higher score as compared to those who did not use it. These higher scores reflected improved accuracy in anticoagulant prescription, anticoagulant management and the diagnosis of VTE. The inclusion of participants with varying levels of education in this study was pre-planned to investigate the impact of experience on VTE management. Our observations revealed a correlation between the scores and years of clinical experience. Hematology fellows obtained the highest scores, followed by internal medicine residents, interns, and medical students. Notably, the mean score increased in all subgroups among participants who utilized the app-based CDSS, contributing to informed decision-making. These findings support the use of the app-based CDSS for all healthcare professionals, particularly benefiting physicians with limited experience in venous thromboembolism (VTE) management.

The randomized control trial evaluating the effectiveness of an artificial intelligence (AI)-CDSS for VTE prophylaxis was conducted in China [8]. The study reported that patients allocated to AI-CDSS were associated with 24% increased rate of mechanical prophylaxis use and a 46% reduction of hospital-associated VTE [8].

The utilization of CDSS to improve anticoagulant management was evaluated in community health care setting in China [9]. Cluster randomization was conducted in two community hospitals. The study reported that patients who were treated in center employing CDSS showed significantly higher rates of guideline-directed antithrombotic therapy as compared to control with an odds ratio of 8.1, 95%CI 2.57–25.34, P < 0.001 [9]. In addition, the incidence of adverse events in CDSS group was lower than that in the control group [9].

A recent systematic review conducted by Sennesael et al. included 16 studies evaluating the computerized CDSS in oral anticoagulant prescription [15]. The review revealed that 9 out of 16 studies showed significant improvement in practitioner performance [15].

There are limited research focusing on an app-based CDSS for the VTE management. Our study indicates that participants expressed satisfaction with the usability and simplicity of the app-based CDSS. In addition, they found that the app-based CDSS was beneficial in providing assistance in decision-making.

There are some limitations in this current study. Firstly, we did not include specialists or other medical professionals, such as clinical pharmacologists or nurse practitioners, which may limit the generalizability of the study findings. Secondly, we did not investigate the benefit of app-based CDSS in the real clinical practice. Consequently, the clinical advantages for VTE patients were not evaluated in this study. Thirdly, the algorithm of the recommendation was based on the ASH guideline. There were minor discrepancies between recent clinical practice guideline on diagnosis and management of VTE. Participants who had experiencing on the other guideline might have alternative management. However, decided to base the clinical decision-making on clinical vignettes, as recommended by the majority of the guidelines. Lastly, the app-based CDSS was focusing on common VTE problems, and we did not include thrombosis in special circumstance, for example, heparin induced thrombocytopenia, anticoagulant management in pregnancy, pediatric thrombosis or thrombosis in unusual sites.

Conclusions

App-based CDSS (DECIDE-COAG^®) demonstrates a significant improvement in the performance of healthcare professionals regarding anticoagulant prescriptions and VTE management. Further studies that specifically investigate the clinical benefits of app-based CDSS in real-world clinical practice are warranted.