Warning: mkdir(): Permission denied in /home/virtual/lib/view_data.php on line 87 Warning: chmod() expects exactly 2 parameters, 3 given in /home/virtual/lib/view_data.php on line 88 Warning: fopen(/home/virtual/e-kjs/journal/upload/ip_log/ip_log_2024-11.txt): failed to open stream: No such file or directory in /home/virtual/lib/view_data.php on line 95 Warning: fwrite() expects parameter 1 to be resource, boolean given in /home/virtual/lib/view_data.php on line 96 Using Machine Learning Models to Identify Factors Associated With 30-Day Readmissions After Posterior Cervical Fusions: A Longitudinal Cohort Study
Neurospine Search

CLOSE


Neurospine > Volume 21(2); 2024 > Article
Gonzalez-Suarez, Rezaii, Herrick, Tigchelaar, Ratliff, Rusu, Scheinker, Jeon, and Desai: Using Machine Learning Models to Identify Factors Associated With 30-Day Readmissions After Posterior Cervical Fusions: A Longitudinal Cohort Study

Abstract

Objective

Readmission rates after posterior cervical fusion (PCF) significantly impact patients and healthcare, with complication rates at 15%–25% and up to 12% 90-day readmission rates. In this study, we aim to test whether machine learning (ML) models that capture interfactorial interactions outperform traditional logistic regression (LR) in identifying readmission-associated factors.

Methods

The Optum Clinformatics Data Mart database was used to identify patients who underwent PCF between 2004–2017. To determine factors associated with 30-day readmissions, 5 ML models were generated and evaluated, including a multivariate LR (MLR) model. Then, the best-performing model, Gradient Boosting Machine (GBM), was compared to the LACE (Length patient stay in the hospital, Acuity of admission of patient in the hospital, Comorbidity, and Emergency visit) index regarding potential cost savings from algorithm implementation.

Results

This study included 4,130 patients, 874 of which were readmitted within 30 days. When analyzed and scaled, we found that patient discharge status, comorbidities, and number of procedure codes were factors that influenced MLR, while patient discharge status, billed admission charge, and length of stay influenced the GBM model. The GBM model significantly outperformed MLR in predicting unplanned readmissions (mean area under the receiver operating characteristic curve, 0.846 vs. 0.829; p < 0.001), while also projecting an average cost savings of 50% more than the LACE index.

Conclusion

Five models (GBM, XGBoost [extreme gradient boosting], RF [random forest], LASSO [least absolute shrinkage and selection operator], and MLR) were evaluated, among which, the GBM model exhibited superior predictive performance, robustness, and accuracy. Factors associated with readmissions impact LR and GBM models differently, suggesting that these models can be used complementarily. When analyzing PCF procedures, the GBM model resulted in greater predictive performance and was associated with higher theoretical cost savings for readmissions associated with PCF complications.

INTRODUCTION

Posterior cervical fusion (PCF) is a common surgical intervention used to treat a variety of cervical spinal pathologies, including spondylosis, spinal tumors, and spinal deformity [1]. However, postoperative complications following PCF are not uncommon; in fact, one literature review reported that patients undergoing PCF have an overall complication rate of 15%–25% [2]. Furthermore, analysis of a spine-specific database for cervical fusion surgeries showed that patients who underwent the posterior approach had unplanned 90-day readmission rates of up to 12% [3]. Such unplanned readmissions contribute to an estimated hospital cost of $10 billion nationally [4]. These costs are increasingly relevant as the rate of cervical fusion surgeries is expected to increase by 13.3% for anterior cervical fusions and 19.3% for PCF [5].
Given these considerations, there is an ongoing effort to generate predictive models that can successfully identify patients at high risk of readmission following spine surgery [6-10]. By identifying high-risk patients, model simulations could offer potential interventions that may reduce readmissions and associated healthcare costs. In particular, machine learning (ML) models have shown promise in identifying intervention strategies that can inform how healthcare and hospital systems allocate resources to reduce postoperative readmission rates [11]. These models leverage patient demographic information and perioperative data to determine relevant factors that predict patient’s risk of being readmitted following surgery. Here, we build on this literature by using ML correlators, including logistic regression (LR) models, to identify risk factors associated with readmissions following PCF. We hypothesized that, in keeping with previously published work, ML models such as least absolute shrinkage and selection operator (LASSO), random forest (RF), stochastic gradient boosting machine (GBM), or extreme gradient boosting (XGBoost) can out predict and outperform traditional LR models, while also contributing to greater readmission-associated cost savings when implemented [12].
To test this 2-fold hypothesis, we (1) compared the predictive performance of 4 supervised ML algorithms to a traditionally-implemented ML model, multivariate LR (MLR) and (2) estimated the potential cost savings of reducing readmissions by implementing the best-performing ML model in a clinical setting and comparing it to the LACE (Length patient stay in the hospital, Acuity of admission of patient in the hospital, Comorbidity, and Emergency visit) index. While this study presents an initial effort to use supervised classification and regression ML models to predict unplanned readmissions rates and simulate readmission-associated costs savings, there are inherent limitations or biases of ML models that are unaccounted for and merit recognition. In addition, further clinical evaluation is needed to refine, finetune, and enhance the performance of the models presented in this study.

MATERIALS AND METHODS

1. Cohort

To analyze specific patient utilization, expenditure, and enrollment data between 1/1/2004 and 11/30/2017, the Optum Clinformatics Data Mart database (Optum, Inc., Eden Prairie, MN, USA) was used. Patients were identified using the Current Procedural Terminology code 22600 for posterior cervical decompression with fusion and 22840, 22842, 22843, or 22844 for posterior spinal instrumentation. Patients were subsequently filtered using our eligibility criteria (Fig. 1). To maintain the specificity of our study population to PCF and instrumentation, we ensured the exclusion of patients who had undergone anterior or lumbar procedures, as these represent distinct surgical categories with different risk profiles and outcomes. Since this database contains deidentified medical claims, patient consent is not applicable, and the study is exempted from requiring Institutional Review Board approval.

2. Predictors

The predictors used were based on previous studies and included patient demographics, socioeconomic status, procedural service, healthcare utilization, complications and comorbidities [13,14] (Table 1). These predictors were selected before analyzing the eligible patient data, and the algorithm implemented to analyze these predictors was in agreement with current medical literature [10-15].

3. Outcomes

The primary outcome assessed was the relative influence of each predictor on the risk for unplanned readmissions, which was normalized using the variable importance score for each model. For secondary outcomes, we measured the performance of each ML model in predicting 30-day readmissions by computing the area under the receiver operating characteristic curve (AUC), which serves as a measure of the model’s ability to estimate the probability of readmission as previously described in Bamber [16]. To calculate the potential cost savings associated with implementing these models, we applied the readmission reduction rates projected by the GBM model to the inflation-adjusted, all-payer national estimates of surgical readmissions within 30 days of surgery, as reported in a previously published Healthcare Cost and Utilization Project Statistical Brief [17]. The brief provides a comprehensive analysis of 30-day post-surgical readmissions and associated costs across a spectrum of high-volume surgeries, including spinal procedures, in various income demographics. The predictive performance and associated cost savings were compared between the top-performing GBM model and the LACE index, a previously-validated readmission model [18].

4. Predictive Modeling

Five models were generated and evaluated: a multivariate LR, a penalized LR model chosen based on elastic net variants of the LASSO, RF, stochastic GBM, and XGBoost. The evaluation of these models was based on AUC, sensitivity, and specificity. The parameters relevant to the prediction task were identified from the variable importance scores, which were calculated for each model and used to improve its interpretability. For model generation and tuning details, refer to the Supplementary Text and Table 1.

5. Statistical Analysis

R ver. 3.5.1 (R Foundation for Statistical Computing, Vienna, Austria) was used to perform all statistical analyses. To determine the risk factors associated with readmissions (Table 1), univariate LR models were generated. To determine the strength of each variable on readmission outcomes, the McFadden Pseudo R2 was computed [19], and the model performance metrics (i.e., AUC, specificity, and sensitivity) were measured using the “caret” statistical package [20]. For all experiments, statistical significance was defined as p < 0.05.

RESULTS

Of the 4,130 patients analyzed in this study, 874 (21.2%) were readmitted within 30 days following surgery. Demographic, medical, socioeconomic, and surgical characteristics were analyzed (Table 1) for all patients who met the selection criteria (Fig. 1). Notably, a univariate LR model identified patient age, insurance plan type, and Medicare use as highly associated with readmissions (p < 0.001). Other variables associated with readmissions were household size and income, presence of an assistant surgeon, number of diagnoses, and length of stay (p < 0.001). Readmitted patients were, on average, older (69.1 years vs. 60.6 years), more likely to have Medicare (81.1% vs. 43.3%), and experienced greater lengths of stay (5.5% vs. 4.8%, p < 0.001). These patients also had greater number of diagnoses (5.7% vs. 4.7%), were significantly more likely to be transferred to a skilled nursing facility (SNF) (63.5% vs. 1.7%), and were more likely to experience postoperative urinary complications (3% vs. 1%, p < 0.001).
Based on the risk factors the univariate LR model associated with readmissions, 5 models were generated and evaluated (GBM, XGBoost, RF, LASSO, and MLR). The performance metrics of these 5 models were measured by using a 50% random sample of the data for training and the remaining 50% for testing (Table 2). GBM outperformed all other models, with an AUC of over 15 independent runs (mean ± standard deviation, 0.844 ± 0.015). Based on these findings, new versions of the GBM (top-performing) and MLR (bottom-performing) models were generated. However, this time, instead of using a 50% random sample of data, all the data from 2014–2016 was used to train the models, and all 2017 cohort data was used to test their performance metrics (Table 3). When comparing the AUC mean values (0.846 vs. 0.829, p < 0.001), the GBM model significantly outperformed the MLR model. The specificity of the GBM model was also superior to that of the MLR model (0.986 vs. 0.966, p < 0.001). The 3 predictors with the greatest relative influence on the GBM model were the discharge status of the patients, the total billed charge of admission, and the length of stay (Fig. 2). Additional factors associated with increased readmissions included the patient’s age, number of prior outpatient visits, as well as the number of procedural and diagnosis codes for initial admission (Fig. 2).
To determine the outcomes of applying interventions to the top 25% of patients with the highest probability of readmission as determined by the GBM model, we subsequently used the GBM model on 2017 model-naive, inflation-adjusted, all-payer national estimates of 30-day surgical readmissions (Table 4). In the 2017 data, we analyzed 490 patients admitted between January and November, of which 133 (27.2%) had been readmitted within 30 days. Of these, the GBM model flagged 105 patients as constituting the top 25% of patients with the highest readmission probability, 99 of which were accurately identified by the model as having been readmitted, giving a true positive rate of 94% (105 top 25% high-risk patients/99 truly readmitted). Out of the remaining 385 patients who were not in the top 25th percentile of high readmission likelihood, 35 patients were nonetheless eventually readmitted, resulting in a missed patient rate of 9% (35 readmitted patients/385 unlikely readmissions). The GBM model predicted an estimated total costs savings of $803,633 over 11 months as a result of reduced readmissions. These cost savings were calculated assuming that 50% of readmissions were prevented by the interventions.
The LACE index was similarly analyzed: in this case, 62 patients out of the 105 total patients (top 25%) with the highest probability of readmission were correctly identified as having been readmitted (true positive rate of 59%). Out of a total of 385 patients that were not included in the top 25th percentile of high readmission likelihood, 73 patients were eventually readmitted (missed patient rate of 19%). The estimated total cost savings associated with reduced readmissions was $535,755 over an 11-month span. Together, this data shows that the GBM model outperformed the LACE index model when comparing the true positive rates (94% vs. 59%), missed patient rates (9% vs. 19%), and the cost savings ($803,633 vs. $535,755). In fact, the GBM model estimated a 50% decrease in readmission-associated costs when compared to those achieved by the LACE index model (Table 4).

DISCUSSION

In order to develop interventions that reduce readmission rates, it is critical to accurately identify patients who are at high risk of being readmitted. One strategy is to identify high-risk patients by analyzing the independent risk factors that are associated with readmission probabilities. In this study, we used epidemiological and supervised ML algorithms to analyze 4,130 patients undergoing PCF. We identified the demographic, socioeconomic, clinical, and procedural characteristics associated with patient readmissions within 30 days.
Univariate LR analysis found that patients’ age, their Medicare usage, their insurance plan type, the number of diagnoses, and length of hospital stay were all variables that influenced the readmission rates (Table 1). Interestingly, patients’ discharge to the SNF was strongly associated with readmissions in both multivariate and univariate LR models. Previous studies have identified similarly significant associations between transfer to SNF and readmission rates; in fact, as many as one in 4 SNF patients experience re-hospitalization within 30 days from their initial admission [7,21,22].
While LR models are commonly used to study and predict unplanned readmission rates, other ML models with the resolution to capture interactions between factors have become popular tools to predict patient outcomes and readmissions [11,13]. A growing body of medical literature has probed the potential of these models in supporting clinical decision-making and implementation [6-10]. Here, we used supervised classification and regression ML algorithms to predict readmissions and identify the risk factors that influence these rates. We found that while GBM identified patients’ discharge status, charge of admission, and length of stay as the most influential predictors of readmissions, MLR identified patients with comorbidities and number of procedure codes as the relevant variables. An explanation for this finding is that LR models make linear predictions on readmissions by computing principled estimates of confidence intervals, while other ML algorithms (i.e., GBM, LASSO, etc.), capture interactions between risk factors and nonlinear relationships. The observed differences in predictor weighting between models shows that, depending on which model is emphasized, it is possible to overestimate or underestimate the relevance of readmission predictors. Thus, by leveraging different ML models, it is possible to capture more realistic linear and nonlinear relationships.
Next, we tested the performance of the ML models, specifically, their ability in predicting 30-day readmissions. Compared to previously reported ML models that predicted readmissions following cervical spine surgery with an AUC mean of 0.63–0.81, our GBM model achieved a mean AUC value of 0.865— the highest predictability performance recorded to date [9,10,23-28]. One explanation for this improved performance is that the variables chosen for the analysis and consequentially, the model’s relative weighting of these variables is unique to this study. For instance, the most influential variables for the GBM model included the discharge destination (i.e., SNF), total charge billed for admission, length of stay, and patient age. To the best of our knowledge, the patient’s discharge destination has been analyzed as a model predictor for 30-day readmission by only one lumbar spine study, which similarly found that the discharge destination was the most influential variable accounted by the model [7].
Next, we used the GBM model to determine the top 25% of patients with the greatest probability of having unplanned readmissions. We then simulated the clinical outcomes of implementing an intervention for these flagged patients. We chose the 25% threshold value to account for hospital systems’ varying capacities to apply interventions for high-risk patients. In practice, however, this threshold should be tuned to the individual capacities and resources of different hospital systems. Of all the patients flagged as having high readmission probabilities, 94% were accurately predicted by the GBM model. The bottom 75% of patients with a high risk for readmissions totaled 385 patients. Of these, only 25 patients were eventually readmitted, accounting for a missed patient rate of 9%. When interventions were simulated for the top 25%, the GBM model presented in this study predicted an estimated cost savings of $803,633 over an 11-month period. It is important to note that this estimated cost savings was computed under the assumption that effective intervention(s) led to half of the high-risk patients not being readmitted.
The findings presented here demonstrate that ML models can identify patients with a high risk of readmission and provide targeted interventions that reduce these patient’s probabilities of being readmitted. While certain identified risk factors, such as age are inherently non-modifiable, the proposed interventions are designed to mitigate the risks associated with modifiable factors of patient’s postoperative care. For instance, while we cannot alter a patient’s age, hospital programs can proactively target those discharged to skilled nursing facilities—a predictor identified by the GBM model for readmissions—by increasing follow-up calls, home visits, and telemonitoring practices, all of which have been shown to reduce readmissions [29-31]. This ensures sustained care and strict adherence to postdischarge protocols, effectively mitigating the risk of readmission. Such measures would be particularly important for complex or invasive procedures, which are often associated with higher billed charges of admission, greater number of procedural and diagnostic codes at outset, and higher number of outpatient visits—all of which are factors that our GBM model identified as predictors of greater patient readmissions. Similarly, patient education is also important in the context of complicated diagnoses and procedures. Empirical evidence suggests that communication interventions at the point of discharge, including medication counseling and disease-specific education, can significantly reduce the likelihood of 30-day readmissions [32]. By integrating structured educational programs that specifically target patients with the highest risks of readmissions, as defined by the GBM model, we can equip patients with the knowledge to manage their conditions more effectively, recognize early signs of complications, and understand when to seek medical help. Last, medication management can address the risks associated with polypharmacy and complex medication regimens often seen in patients with multiple codes for procedures and diagnoses. A pharmacist-led approach, encompassing medication reconciliation, a patient-specific medication care plan, discharge counseling, and follow-up contact, can substantially decrease the incidence of medication errors postdischarge, thereby lessening the chances of readmission or emergency department visits [33]. By using ML models to identify patients at high risk of readmission and targeting these patients specifically, resources may be optimally allocated while contributing to a reduction in the costs associated with readmitting these high-risk patients. Nonetheless, future research could benefit from a closer examination of the direct impact of these interventions on readmission rates.
Both our cost savings simulation and previously reported data provide evidence that reducing the number of readmissions, even slightly, can significantly lower the economic burden of unplanned readmissions [34]. With further fine-tuning and customization, these models could (1) aid clinicians in identifying patients with high readmission risks; (2) guide perioperative resource allocation to decrease readmission probabilities; and (3) decrease overall healthcare costs.
In addition to comparing the GBM model to univariate and multivariate LR models, we also analyzed how the performance of our ML algorithm compared to that of clinically-employed predictive models of readmission (i.e., the LACE index model). When we analyzed the top 25% of patients with a high risk for unplanned readmissions, we found that in comparison to the LACE index model, the GBM had a higher true positive rate of readmission, a 50% higher cost savings from readmission prevention, and a lower missed patient rate. Several factors may explain GBM’s higher performance when compared to the LACE index model. For instance, we provided the ML model with an expanded set of predictors to utilize and GBM has an adaptive learning framework that can capture and leverage interactions between variables, including nonlinear relationships. Currently, there is no consensus on the performance power of the LACE index; while some reports strongly support the model’s ability to discriminate between variables [35-37], others cite the lack of strong predictors [38] as reason for the model’s moderate to poor discrimination [38-41].
Our study generated models that capture patients at high risk of readmissions and simulates how hypothetical clinical interventions can reduce healthcare utilizations costs. However, there are several limitations worth noting when interpreting these results. First, the insurance claims database used was not specific to spine surgery, which can impact accuracy [42,43]. Second, we must understand the limitations of using administrative claims data to guide and alter clinical practice [42,43]. For example, during the study period, insurers and hospitals attempted to decrease patient’s length of stay by discharging to home. This creates confounds in the associations captured between readmission probabilities and patient’s length of stay. Third, to simulate how intervening of high-risk patients would impact healthcare utilization, we assumed that 50% of the theoretical interventions were effective, which could vary by hospital systems.
These limitations highlight substantial avenues for improvement. Although our study represents an initial effort to utilize ML models to predict readmission rates and mitigate healthcare costs, future prospective studies and clinical trials are needed to refine, validate, and enhance the ML algorithms presented. Moreover, exploring the utilization of alternative data sources beyond administrative claims (i.e., prospective datasets) and predicting 90 days in addition to 30-day readmission rates, could further enhance the robustness and generalizability of our findings.

CONCLUSION

ML models, including MLR, have identified different risk factors associated with patients’ unplanned readmission probabilities. This is potentially a result of each model’s capacity to measure nonlinear relationships and interfactor interactions, and it suggests that models can complement each other when capturing risk predictors for readmissions following PCF. In addition, we found that when comparing 2 ML models, GBM outperformed the MLR model as measured by the mean AUC. These findings support the rationale for the continued generation, improvement, and eventual implementation of ML models in order to reduce readmissions and associated healthcare utilization costs.

Supplementary Material

Supplementary Text and Table 1 can be found via https://doi.org/10.14245/ns.2347340.670.
Supplementary Text.
Data preprocessing, model generation, and model tuning
ns-2347340-670-Supplementary.pdf

NOTES

Conflict of Interest

The authors have nothing to disclose.

Funding/Support

This study received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Author Contribution

Conceptualization: ADGS, PGR, DH, JKR, MR, DS, AMD; Data curation: PGR; Formal analysis: PGR; Methodology: PGR, DH, JKR, MR, DS, AMD; Project administration: ADGS, JKR, MR, DS, IJ, AMD; Visualization: ADGS; Writing – original draft: ADGS; Writing – review & editing: ADGS, PGR, SST, IJ, AMD.

Fig. 1.
Identifying and selecting criteria for eligible patients who underwent posterior cervical fusion and instrumentation. “Complete cases” refers to patients with no missing values for model predictors.
ns-2347340-670f1.jpg
Fig. 2.
Graph depicts the relative weighting of various risk factors for 30-day readmissions, as determined by a Gradient Boosting Machine (GBM) machine learning model. The y-axis lists the risk factors, including both linear terms and polynomial transformations of certain variables like “Year of surgery” and “Household income.” The x-axis displays the relative weightings of each risk factor, scaled against the highest weighted risk factor, which was “Transfer to skilled nursing facility” (not shown in this part of the chart). Each bar’s length indicates the strength of its association with 30-day readmissions, where a longer bar signifies a stronger predictor. ED, Emergency Department.
ns-2347340-670f2.jpg
Table 1.
Univariate logistic regression
Variable Not readmitted Readmitted Coefficient (SE) Pseudo R2 p-value
Demographics
 Age (yr) 60.6 ± 10.9* 69.1 ± 10.6* 0.077 (0.004) 0.098 < 0.001
 Female sex 1,475 (45.3) 397 (45.4) 0.005 (0.077) < 0.001 0.949
 Medicare 1,411 (43.3) 709 (81.1) 1.726 (0.093) 0.099 < 0.001
 Geographic division 0.011 < 0.001
  South Atlantic 895 (27.5) 209 (23.9)
  East North Central 550 (16.9) 174 (19.9) 0.304 (0.116)
  West South Central 398 (12.2) 97 (11.1) 0.043 (0.137)
  Mountain 325 (10.0) 76 (8.7) 0.001 (0.149)
  West North Central 359 (11.0) 66 (7.6) -0.239 (0.154)
  Pacific 283 (8.7) 93 (10.6) 0.342 (0.142)
  Middle Atlantic 176 (5.4) 75 (8.6) 0.602 (0.158)
  New England 132 (4.1) 60 (6.9) 0.666 (0.174)
  East South Central 138 (4.2) 24 (2.7) -0.295 (0.234)
 Insurance 0.083 < 0.001
  POS 1,237 (38.0) 105 (12.0)
  HMO 760 (23.3) 271 (31.0) 1.435 (0.124)
  Other 753 (23.1) 401 (45.9) 1.836 (0.119)
  PPO 227 (7.0) 75 (8.6) 1.359 (0.168)
  EPO 219 (6.7) 17 (1.9) -0.089 (0.272)
  Indemnity 60 (1.8) 5 (0.6) -0.018 (0.476)
Comorbidities
 Alcohol 41 (1.3) 12 (1.4) 0.088 (0.330) < 0.001 0.792
 Anemia 111 (3.4) 38 (4.3) 0.253 (0.192) < 0.001 0.196
 Arrhythmia 225 (6.9) 80 (9.2) 0.306 (0.136) 0.001 0.028
 CHF 67 (2.1) 38 (4.3) 0.772 (0.207) 0.003 < 0.001
 Coagulopathy 24 (0.7) 15 (1.7) 0.855 (0.331) 0.001 0.014
 Depression 454 (13.9) 104 (11.9) -0.182 (0.116) 0.001 0.112
 DM 525 (16.1) 178 (20.4) 0.286 (0.097) 0.002 0.004
 DMcx 97 (3.0) 56 (6.4) 0.802 (0.172) 0.005 < 0.001
 Drugs 26 (0.8) 15 (1.7) 0.774 (0.327) 0.001 0.023
 Fluid/electrolyte disorder 163 (5.0) 80 (9.2) 0.648 (0.142) 0.005 < 0.001
 HTN 1,619 (49.7) 477 (54.6) 0.195 (0.076) 0.002 0.011
 HTNcx 62 (1.9) 39 (4.5) 0.878 (0.208) 0.004 < 0.001
 Hypothyroid 323 (9.9) 73 (8.4) -0.189 (0.136) 0.001 0.156
 Liver 34 (1.0) 10 (1.1) 0.092 (0.362) < 0.001 0.800
 Lymphoma 11 (0.3) 1 (0.1) -1.085 (1.045) < 0.001 0.228
 Metastatic disease 11 (0.3) 8 (0.9) 1.003 (0.466) 0.001 0.040
 Neurologic (other) 82 (2.5) 50 (5.7) 0.854 (0.184) 0.005 < 0.001
 Obesity 326 (10.0) 104 (11.9) 0.194 (0.120) 0.001 0.110
 Paralysis 60 (1.8) 47 (5.4) 1.108 (0.199) 0.007 < 0.001
 PHTN 7 (0.2) 10 (1.1) 1.681 (0.494) 0.003 0.001
 Psychoses 40 (1.2) 14 (1.6) 0.269 (0.313) < 0.001 0.400
 Pulmonary 572 (17.6) 151 (17.3) -0.020 (0.101) < 0.001 0.841
 PVD 65 (2.0) 34 (3.9) 0.687 (0.215) 0.002 0.002
 Renal 106 (3.3) 71 (8.1) 0.966 (0.158) 0.008 < 0.001
 Rheumatic 175 (5.4) 39 (4.5) -0.196 (0.181) < 0.001 0.272
 Tobacco 775 (23.8) 160 (18.3) -0.332 (0.097) 0.003 < 0.001
 Tumor 114 (3.5) 26 (3.0) -0.168 (0.221) < 0.001 0.438
 Valvular 66 (2.0) 24 (2.7) 0.311 (0.242) < 0.001 0.209
 Weight loss 8 (0.2) 8 (0.9) 1.322 (0.502) 0.002 0.011
Medical characteristics
 Transferred to SNF 54 (1.7) 555 (63.5) 4.636 (0.154) 0.412 < 0.001
 Diagnoses (n) 9.9 ± 4.7 11 ± 5.7 0.045 (0.007) 0.009 < 0.001
 Length of stay (nights) 4.1 ± 4.8 6.5 ± 5.5 0.084 (0.008) 0.031 < 0.001
 Prior admissions (n) 0.4 ± 0.7 0.8 ± 1.1 0.520 (0.045) 0.033 < 0.001
 Prior ED visits (n) 0.5 ± 1.8 0.9 ± 1.8 0.098 (0.018) 0.007 < 0.001
 Prior major operations (n) 0.3 ± 0.6 0.3 ± 0.6 0.066 (0.064) < 0.001 0.307
 Prior outpatient visits (n) 17.3 ± 11.4 21.2 ± 14.9 0.022 (0.003) 0.014 < 0.001
 Primary diagnosis 0.012 < 0.001
  Stenosis 1,135 (34.9) 341 (39.0)
  Spondylolisthesis 997 (30.6) 305 (34.9) 0.018 (0.090)
  DDD 588 (18.1) 118 (13.5) -0.404 (0.118)
  Displacement 206 (6.3) 63 (7.2) 0.018 (0.157)
  Spondylosis 130 (4.0) 9 (1.0) -1.468 (0.350)
  Other 88 (2.7) 9 (1.0) -1.078 (0.355)
  Radiculopathy 50 (1.5) 15 (1.7) -0.002 (0.301)
  Pain 33 (1.0) 8 (0.9) -0.215 (0.399)
  Deformity 23 (0.7) 5 (0.6) -0.324 (0.497)
  Postlaminectomy syndrome 4 (0.1) 1 (0.1) -0.184 (1.120)
  Spondylolysis 2 (0.1) 0 (0) -11.364 (229.629)
Socioeconomic characteristics
 Education 0.002 0.018
  < 12th grade 16 (0.5) 3 (0.3)
  High school diploma 1,028 (31.6) 323 (37.0) 0.074 (0.430)
  Some college 1,803 (55.4) 456 (52.2) -0.317 (0.323)
  Bachelor’s degree 409 (12.6) 92 (10.5) 0.186 (0.153)
 Household size 0.063 < 0.001
  1 1,637 (50.3) 698 (79.9)
  2 870 (26.7) 107 (12.2) -1.128 (0.328)
  3 290 (8.9) 26 (3.0) 0.621 (0.313)
  4 269 (8.3) 23 (2.6) -0.698 (0.259)
  5 128 (3.9) 16 (1.8) -0.153 (0.220)
  6+ 62 (1.9) 4 (0.5) -0.154 (0.211)
 Household income (USD) 0.022 < 0.001
  < 40,000 781 (24.0) 341 (39.0)
  40,000–49,999 260 (8.0) 57 (6.5) -0.591 (0.092)
  50,000–59,999 255 (7.8) 84 (9.6) 0.05 (0.095)
  60,000–74,999 422 (13.0) 99 (11.3) -0.229 (0.112)
  75,000–99,999 571 (17.5) 123 (14.1) 0.278 (0.120)
  > 100,000 967 (29.7) 170 (19.5) -0.266 (0.120)
 Race 0.005 0.001
  White 2,579 (79.2) 628 (71.9)
  Black 406 (12.5) 148 (16.9) 0.404 (0.106)
  Hispanic 225 (6.9) 79 (9.0) 0.366 (0.138)
  Asian 46 (1.4) 19 (2.2) 0.528 (0.276)
Surgical characteristics
 Microscope 202 (6.2) 28 (3.2) -0.692 (0.205) 0.003 < 0.001
 Navigation 210 (6.4) 59 (6.8) 0.049 (0.153) < 0.001 0.750
 rhBMP 251 (7.7) 52 (5.9) -0.278 (0.157) 0.001 0.070
 Allograft 1,034 (31.8) 263 (30.1) -0.078 (0.083) < 0.001 0.350
 Assistant surgeon 1,247 (38.3) 152 (17.4) -1.081 (0.096) 0.034 < 0.001
 Increased procedural service 27 (0.8) 9 (1.0) 0.219 (0.387) < 0.001 0.579
 Levels
  2 452 (13.9) 67 (7.7) 0.011 < 0.001
  3–6 2,493 (76.6) 670 (76.7) 1.384 (0.370)
  7–12 304 (9.3) 130 (14.9) 0.127 (0.281)
  13+ 7 (0.2) 7 (0.8) 0.115 (0.145)
 Multiple procedures 409 (12.6) 194 (22.2) 0.686 (0.097) 0.011 < 0.001
 Charge (USD) 121,618.6 ± 129,387.3 173,765.3 ± 248,002.8 0 (0) 0.015 < 0.001
 Complications (n) 0.3 ± 0.8 0.4 ± 0.8 0.041 (0.047) < 0.001 0.396
 CPT codes (n) 11.6 ± 4.0 12.2 ± 4.0 0.040 (0.009) 0.004 < 0.001
 Return to OR 101 (3.1) 39 (4.5) 0.378 (0.193) 0.001 0.056
 Year of surgery 0.023 < 0.001
  2004 50 (1.5) 4 (0.5)
  2005 64 (2.0) 8 (0.9) 1.958 (0.304)
  2006 119 (3.7) 8 (0.9) -0.311 (0.291)
  2007 124 (3.8) 13 (1.5) -0.321 (0.287)
  2008 191 (5.9) 32 (3.7) 0.134 (0.280)
  2009 247 (7.6) 40 (4.6) 0.042 (0.259)
  2010 255 (7.8) 55 (6.3) -0.054 (0.244)
  2011 272 (8.4) 63 (7.2) 0.395 (0.239)
  2012 264 (8.1) 68 (7.8) -0.312 (0.231)
  2013 320 (9.8) 104 (11.9) 0.193 (0.226)
  2014 271 (8.3) 102 (11.7) -0.175 (0.214)
  2015 273 (8.4) 95 (10.9) -0.017 (0.188)
  2016 395 (12.1) 130 (14.9) 0.115 (0.166)
  2017 411 (12.6) 152 (17.4) -0.077 (0.151)
Complications
 Acute renal failure 20 (0.6) 15 (1.7) 1.039 (0.344) 0.002 0.004
 Orthopedic implant 299 (9.2) 36 (4.1) -0.856 (0.181) 0.006 < 0.001
 Cardiovascular 24 (0.7) 4 (0.5) -0.479 (0.541) < 0.001 0.349
 Dural tear 23 (0.7) 10 (1.1) 0.487 (0.381) < 0.001 0.217
 Neurologic 21 (0.6) 6 (0.7) 0.063 (0.465) < 0.001 0.893
 Pulmonary 92 (2.8) 41 (4.7) 0.526 (0.192) 0.002 0.008
 Wound (other) 6 (0.2) 3 (0.3) 0.624 (0.708) < 0.001 0.398
 Pneumonia 11 (0.3) 5 (0.6) 0.529 (0.541) < 0.001 0.346
 Sepsis or septicemia 0 (0) 3 (0.3) 13.885 (187.491) 0.002 0.002
 Thrombotic 9 (0.3) 3 (0.3) 0.217 (0.668) < 0.001 0.750
 Urinary tract infection 32 (1.0) 26 (3.0) 1.128 (0.267) 0.004 < 0.001
 Wound hematoma/hemorrhage 14 (0.4) 12 (1.4) 1.171 (0.395) 0.002 0.004
 Any 501 (15.4) 138 (15.8) 0.031 (0.105) < 0.001 0.771

Values are presented as number (%) or mean±standard deviation unless otherwise indicated.

SE, standard error; POS, place of service; HMO, health maintenance organization; PPO, preferred provider organization; EPO, exclusive provider network; CHF, congestive heart failure; DM, diabetes mellitus; DMcx, complicated diabetes mellitus; HTN, hypertension; HTNcx, complicated hypertension; PHTN, pulmonary hypertension; PVD, peripheral vascular disease; SNF, skilled nursing facility; ED, Emergency Department; DDD, degenerative disc disease; USD, United States dollar; rhBMP, recombinant human bone morphogenetic protein; CPT, current procedural terminology; OR, operating room.

Table 2.
Performance metrics of models generated using 50% train and 50% test data splits
Model AUC
GBM 0.844 ± 0.015
XGBoost 0.842 ± 0.016
RF 0.843 ± 0.015
LASSO 0.833 ± 0.021
LR 0.819 ± 0.021

Values are presented as mean±standard deviation.

AUC, area under the receiver operating characteristic curve; GBM, Gradient Boosting Machine; XGBoost, extreme gradient boosting; RF, random forest; LASSO, least absolute shrinkage and selection operator; LR, traditional logistic regression.

Table 3.
Performance metrics of GBM and LR models generated using data from 2004–2016 for training and data from 2017 for testing
Variable GBM LR Difference 95% CI p-value
AUC 0.846 ± 0.016 0.829 ± 0.018 0.0170 0.012–0.022 < 0.001
Sensitivity 0.610 ± 0.033 0.625 ± 0.037 0.0152 0.021–0.010 < 0.001
Specificity 0.986 ± 0.002 0.966 ± 0.009 0.0195 0.014–0.025 < 0.001

Values are presented as mean±standard deviation unless otherwise indicated.

GBM, Gradient Boosting Machine; LR, traditional logistic regression; CI, confidence interval; AUC, area under the receiver operating characteristic curve.

Table 4.
Predictive performance of GBM and LACE index on 2017 test data and associated cost savings
Month Top 25% highest risk patients (n) True positive rate on top 25%
Bottom 75% risk patients (n) Missed patients rate
Estimated cost savings* using GBM (USD) Average cost savings using GBM over LACE
GBM LACE GBM LACE
January 12 1.00 0.60 38 0.03 0.16 97,410 100%
February 5 1.00 0.60 18 0.17 0.26 40,588 67%
March 12 1.00 0.70 36 0.22 0.35 97,410 71%
April 11 1.00 0.80 36 0.17 0.26 89,293 38%
May 4 1.00 0.38 27 0.04 0.11 32,470 33%
June 5 1.00 0.83 21 0.14 0.24 40,588 0%
July 7 1.00 0.67 33 0.09 0.13 56,823 17%
August 11 1.00 0.64 39 0.10 0.24 89,293 57%
September 6 1.00 0.50 39 0.05 0.09 48,705 20%
October 13 0.85 0.71 48 0.15 0.16 89,293 10%
November 12 0.83 0.43 52 0.06 0.14 81,175 67%
All 105 0.94 0.59 385 0.09 0.19 803,633 50%

GBM, Gradient Boosting Machine; LACE, Length patient stay in the hospital, Acuity of admission of patient in the hospital, Comorbidity, and Emergency visit; USD, United States dollar.

* Cost savings were calculated under the assumption that 50% of readmissions were prevented via targeted interventions for patients correctly identified by the model as likely for readmission.

REFERENCES

1. Youssef JA, Heiner AD, Montgomery JR, et al. Outcomes of posterior cervical fusion and decompression: a systematic review and meta-analysis. Spine J 2019;19:1714-29.
crossref pmid
2. Badiee RK, Mayer R, Pennicooke B, et al. Complications following posterior cervical decompression and fusion: a review of incidence, risk factors, and prevention strategies. J Spine Surg 2020;6:323-33.
crossref pmid pmc
3. Schafer E, Bazydlo M, Schultz L, et al. Rates and risk factors associated with 90-day readmission following cervical spine fusion surgery: analysis of the Michigan Spine Surgery Improvement Collaborative (MSSIC) registry. Spine J 2020;20:708-16.
crossref pmid
4. Martin BI, Mirza SK, Spina N, et al. Trends in lumbar fusion procedure rates and associated hospital costs for degenerative spinal diseases in the united states, 2004 to 2015. Spine (Phila Pa 1976) 2019;44:369-76.
crossref pmid
5. Neifert SN, Martini ML, Yuk F, et al. Predicting trends in cervical spinal surgery in the United States from 2020 to 2040. World Neurosurg 2020;141:e175-81.
crossref pmid
6. Hopkins BS, Yamaguchi JT, Garcia R, et al. Using machine learning to predict 30-day readmissions after posterior lumbar fusion: an NSQIP study involving 23,264 patients. J Neurosurg Spine November 2019;32:399-406.
crossref
7. Kalagara S, Eltorai AEM, Durand WM, et al. Machine learning modeling for predicting hospital readmission following lumbar laminectomy. J Neurosurg Spine 2018;30:344-52.
crossref pmid
8. McGirt MJ, Sivaganesan A, Asher AL, et al. Prediction model for outcome after low-back surgery: individualized likelihood of complication, hospital readmission, return to work, and 12-month improvement in functional disability. Neurosurg Focus 2015;39:E13.
crossref
9. Parker SL, Sivaganesan A, Chotai S, et al. Development and validation of a predictive model for 90-day readmission following elective spine surgery. J Neurosurg Spine 2018;29:327-31.
crossref pmid
10. Goyal A, Ngufor C, Kerezoudis P, et al. Can machine learning algorithms accurately predict discharge to nonhome facility and early unplanned readmissions following spinal fusion? Analysis of a national surgical registry. J Neurosurg Spine 2019;31:568-78.
crossref pmid
11. Artetxe A, Beristain A, Graña M. Predictive models for hospital readmission risk: a systematic review of methods. Comput Methods Programs Biomed 2018;164:49-64.
crossref pmid
12. Rezaii PG, Herrick D, Ratliff JK, et al. Identification of factors associated with 30-day readmissions after posterior lumbar fusion using machine learning and traditional models: a national longitudinal database study. Spine (Phila Pa 1976) 2023;48:1224-33.
pmid
13. Elixhauser A, Steiner C, Harris DR, et al. Comorbidity measures for use with administrative data. Med Care 1998;36:8-27.
crossref pmid
14. Quan H, Li B, Couris CM, et al. Updating and validating the Charlson comorbidity index and score for risk adjustment in hospital discharge abstracts using data from 6 countries. Am J Epidemiol 2011;173:676-82.
crossref pmid
15. Hasan O, Meltzer DO, Shaykevich SA, et al. Hospital readmission in general medicine patients: a prediction model. J Gen Intern Med 2010;25:211-9.
crossref pmid pmc pdf
16. Bamber D. The area above the ordinal dominance graph and the area below the receiver operating characteristic graph. J Math Psychol 1975;12:387-415.
crossref
17. Qasim M, Andrews RM. Post-surgical readmissions among patients living in the poorest communities, 2009. Healthcare Cost and Utilization Project (HCUP) statistical briefs. Rockville (MD): Agency for Healthcare Research and Quality (US); 2006.

18. Akins PT, Harris J, Alvarez JL, et al. Risk factors associated with 30-day readmissions after instrumented spine surgery in 14,939 patients: 30-day readmissions after instrumented spine surgery. Spine (Phila Pa 1976) 2015;40:1022-32.
pmid
19. Hauber AB, González JM, Groothuis-Oudshoorn CGM, et al. Statistical methods for the analysis of discrete choice experiments: a report of the ISPOR conjoint analysis good research practices task force. Value Health 2016;19:300-15.
crossref pmid
20. Kuhn M. Building predictive models in R using the caret package. J Stat Softw 2008;28:1-26.
pmid pmc
21. Kim LD, Pfoh ER, Hu B, et al. Derivation and validation of a model to predict 30-day readmission in surgical patients discharged to skilled nursing facility. J Am Med Dir Assoc 2019;20:1086-90.e2.
crossref pmid
22. Mor V, Intrator O, Feng Z, et al. The revolving door of rehospitalization from skilled nursing facilities. Health Aff (Millwood) 2010;29:57-64.
crossref pmid pmc
23. Devin CJ, Bydon M, Alvi MA, et al. A predictive model and nomogram for predicting return to work at 3 months after cervical spine surgery: an analysis from the Quality Outcomes Database. Neurosurg Focus 2018;45:E9.
crossref
24. Merali ZG, Witiw CD, Badhiwala JH, et al. Using a machine learning approach to predict outcome after surgery for degenerative cervical myelopathy. PLoS One 2019;14:e0215133.
crossref pmid pmc
25. Asher AL, Devin CJ, Kerezoudis P, et al. Predictors of patient satisfaction following 1- or 2-level anterior cervical discectomy and fusion: insights from the Quality Outcomes Database. J Neurosurg Spine 2019;31:835-43.
crossref pmid
26. Siccoli A, de Wispelaere MP, Schröder ML, et al. Machine learning-based preoperative predictive analytics for lumbar spinal stenosis. Neurosurg Focus 2019;46:E5.
crossref
27. Lopez CD, Boddapati V, Lombardi JM, et al. Artificial learning and machine learning applications in spine surgery: a systematic review. Global Spine J 2022;12:1561-72.
crossref pmid pmc pdf
28. Lubelski D, Hersh A, Azad TD, et al. Prediction models in degenerative spine surgery: a systematic review. Global Spine J 2021;11(1_suppl):79S-88S.
crossref pmid pmc pdf
29. Stewart S, Pearson S, Horowitz JD. Effects of a home-based intervention among patients with congestive heart failure discharged from acute hospital care. Arch Intern Med 1998;158:1067-72.
crossref pmid
30. Dudas V, Bookwalter T, Kerr KM, et al. The impact of follow-up telephone calls to patients after hospitalization. Am J Med 2001;111(9B):26S-30S.
crossref pmid
31. Dawson NL, Hull BP, Vijapura P, et al. Home telemonitoring to reduce readmission of high-risk patients: a modified intention-to-treat randomized clinical trial. J Gen Intern Med 2021;36:3395-401.
crossref pmid pmc pdf
32. Becker C, Zumbrunn S, Beck K, et al. Interventions to improve communication at hospital discharge and rates of readmission: a systematic review and meta-analysis. JAMA Netw Open 2021;4:e2119346.
crossref pmid pmc
33. Phatak A, Prusi R, Ward B, et al. Impact of pharmacist involvement in the transitional care of high-risk patients through medication reconciliation, medication education, and postdischarge call-backs (IPITCH Study). J Hosp Med 2016;11:39-44.
pmid
34. Sweeney JF. Postoperative complications and hospital readmissions in surgical patients: an important association. Ann Surg 2013;258:19-20.
pmid
35. Garrison GM, Robelia PM, Pecina JL, et al. Comparing performance of 30-day readmission risk classifiers among hospitalized primary care patients. J Eval Clin Pract 2017;23:524-9.
crossref pmid pdf
36. Spiva L, Hand M, VanBrackle L, et al. Validation of a predictive model to identify patients at high risk for hospital readmission. J Healthc Qual 2016;38:34-41.
crossref pmid
37. van Walraven C, Dhalla IA, Bell C, et al. Derivation and validation of an index to predict early death or unplanned readmission after discharge from hospital to the community. CMAJ 2010;182:551-7.
crossref pmid pmc
38. Cotter PE, Bhalla VK, Wallis SJ, et al. Predicting readmissions: poor performance of the LACE index in an older UK population. Age Ageing 2012;41:784-9.
crossref pmid
39. Tong L, Erdmann C, Daldalian M, et al. Comparison of predictive modeling approaches for 30-day all-cause non-elective readmission risk. BMC Med Res Methodol 2016;16:26.
crossref pmid pmc
40. Predicting 30-day readmissions: performance of the LACE index compared with a regression model among general medicine patients in Singapore. Biomed Res Int 2015;2015:169870.
pmid pmc
41. Robinson R, Hudali T. The HOSPITAL score and LACE index as predictors of 30 day readmission in a retrospective study at a university-affiliated community hospital. PeerJ 2017;5:e3137.
crossref pmid pmc pdf
42. Kestle JR. Administrative database research. J Neurosurg 2015;122:441-2.
pmid
43. Oravec CS, Motiwala M, Reed K, et al. Big data research in neurosurgery: a critical look at this popular new study design. Neurosurgery 2018;82:728-46.
crossref pmid


Editorial Office
Department of Neurosurgery, CHA Bundang Medical Center,
CHA University School of Medicine,
59 Yatap-ro, Bundang-gu, Seongnam 13496, Korea
Tel: +82-31-780-1924  Fax: +82-31-780-5269  E-mail: support@e-neurospine.org
The Korean Spinal Neurosurgery Society
#407, Dong-A Villate 2 Town, 350 Seocho-daero, Seocho-gu, Seoul 06631, Korea
Tel: +82-2-585-5455  Fax: +82-2-2-523-6812  E-mail: ksns1987@gmail.com
Business License No.: 209-82-62443

Copyright © The Korean Spinal Neurosurgery Society.

Developed in M2PI

Zoom in Close layer