Nephrol Dial Transplant (2003) 18: 77-81
© 2003 European Renal Association-European Dialysis and Transplant Association
Prediction of acute renal failure after cardiac surgery: retrospective cross-validation of a clinical algorithm
1 Section of Nephrology, Department of Medicine, University Hospital of North Norway, 2 Faculty of Medicine, University of Tromsø and 3 Department of Thoracic and Cardiovascular Surgery, University Hospital of North Norway, Tromsø, Norway
 |
Abstract |
|---|
 Top Abstract IntroductionSubjects and methodsResultsDiscussionReferences |
|---|
Background. Acute renal failure (ARF) after cardiac surgery is associated with high costs and a poor prognosis. Based on the results of a large US study, an algorithm has been developed for predicting ARF from pre-operative risk factors. The aim of this study was to cross-validate this algorithm in a patient population from Europe, and to assess its usefulness as a clinical tool.
Methods. All coronary bypass and valvular surgery patients from a 5-year period were included. Data on pre-operative risk factors for all patients who developed dialysis-dependent ARF and for a random sample of patients without ARF were retrospectively obtained from hospital databases and medical records. For each patient, a risk score for ARF was calculated on the basis of the algorithm. The sensitivity, specificity, positive and negative predictive values and area under receiver operating characteristic (ROC) curve of the score's ability to predict ARF were estimated.
Results. 2037 patients were included. The risk of ARF was 1.9%, and the area under the ROC curve 0.71. For a risk score of 6 or higher, the sensitivity was 0.53, the specificity 0.71, the positive predictive value 0.03 and the negative predictive value 0.99.
Conclusions. The validity of the algorithm was confirmed in a population differing in several aspects from the US populations where it was developed. Although it is useful for estimating the risk of ARF in groups of patients, the low risk of ARF limits the algorithm's ability to predict the outcome for individual patients.
Keywords: cardiac surgical procedures; coronary artery bypass; kidney failure acute; post-operative complications; receiver operating characteristic curve; sensitivity and specificity
|
Introduction |
|---|
|
TopAbstractIntroductionSubjects and methodsResultsDiscussionReferences |
|---|
Acute renal failure (ARF) is a serious complication of cardiac surgery. In different studies, the risk of developing a need for dialysis in the post-operative period has varied between 0.5 and 15% [1–9]. The prognosis of ARF in this setting is poor, with mortality rates ranging from 28 to 64% [2,3,5,7,8]. These patients often have long stays in intensive care units and high costs [10].
Changes in the treatment of post-operative ARF have had little impact on the outcome, possibly because improvements in therapy have been balanced by increasing pre-operative comorbidity [11]. For this reason, the possibility of identifying patients with a high risk of ARF before surgery has been investigated. In 1997, Chertow et al. [3] published an algorithm for predicting ARF from pre-operative data based on a study of patients in the Veterans Administration Continuous Improvement in Cardiac Surgery Program (CICSP). The algorithm estimates the risk of ARF from a severity score obtained by adding points for separate risk factors. It was validated in an independent sample of patients and demonstrated good discrimination. In a later study, Fortescue et al. [4] validated the algorithm in the Quality Measurement and Management Initiative (QMMI) cohort. Again, the algorithm performed well in discriminating for the development of ARF.
Cross-validation of risk-stratification algorithms is necessary before they can be accepted as useful in clinical practice [12,13]. So far, the CICSP algorithm has only been validated in patients in the US in the first half of the 1990s. The aim of the present study was to validate the algorithm in patients operated on at the University Hospital of North Norway between 1995 and 2000. Also, we wanted to compare risk factors for ARF after cardiac surgery with those found in the US cohorts and to assess the algorithm's usefulness by estimating the positive and negative predictive values of its predictions of ARF.
|
Subjects and methods |
|---|
|
TopAbstractIntroductionSubjects and methodsResultsDiscussionReferences |
|---|
Subjects
All patients having undergone coronary artery bypass grafting and/or valvular surgery at the University Hospital of North Norway between the 1 January 1995 and the 31 December 1999 were identified retrospectively from the hospital's registers. Patients who had been operated more than once in the period were considered to be at risk for each new operation, except for re-operations taking place within 30 days of the primary operation. Patients who had simultaneously been subjected to other surgical procedures, who were receiving renal replacement therapy or who had active endocarditis at the time of operation were excluded. In order to be consistent with the CICSP study, patients with a serum creatinine >265 mmol/l (3 mg/dl) pre-operatively were also excluded [3,4].
Patients in need of renal replacement therapy within 30 days of the operation were considered to have experienced ARF as a complication. These patients were identified by review of medical records. A 33% random sample of the patients without ARF was drawn as controls, but because of financial limitations only 32% were actually examined. These 32% still constitute a valid random sample, and no bias was introduced by omission of the last 1%. Randomization was performed with a pseudo-random number generator.
Data
Age, gender and serum creatinine were obtained directly from hospital databases. Other data necessary to implement the CICSP algorithm were extracted from medical records by a medical student (K.R.S.H.) under the supervision of a nephrologist (B.O.E.). As in the validation based on QMMI data, left-ventricular ejection fraction <35% was used as a proxy for cardiomegaly [4]. For the other variables in the algorithm, we applied the definitions of risk factors from the CICSP study [3].
Creatinine clearance was estimated from serum creatinine, age, weight and gender by the formula of Cockcroft and Gault [14].
Operative mortality was defined as death from any cause within 30 days of surgery.
Statistical methods
A multivariate logistic regression analysis with ARF as the dichotomous dependent variable and the risk factors of the CICSP algorithm as the independent variables was performed with the SAS statistical package (SAS Institute, Cary, NC). Calibration of the model was assessed using the Hosmer–Lemeshow goodness-of-fit test [15].
The CICSP additive severity score was calculated for each patient by adding points for each risk factor present, as specified in the study of Fortescue et al. [4]. The points associated with each type of risk factor were based on the odds-ratios (OR) found in the logistic regression analysis of risk factors in the CICSP study [3].
The score's sensitivity, specificity and positive and negative predictive value for predicting ARF for different cut-off levels of the score were calculated according to standard formulae as follows [16]:
 |
 |
 |
 |
For the calculation of the predictive values, the number of patients without ARF with a given score was estimated from the number of those randomly drawn by dividing this number by 0.32, i.e. the proportion of randomly drawn patients without ARF.
A receiver operating characteristic curve (ROC curve) was constructed by plotting the sensitivity vs (1 – specificity) for all possible cut-off levels. The ROC curve area was calculated by the non-parametric technique of Hanley and McNeill [17].
Ninety-five per cent confidence intervals (CI) of proportions were calculated as exact binomial CI.
Missing data for dichotomous variables were coded as absent, and average values were substituted for missing continuous variables. Analyses were also repeated without patients with missing information.
Statistical significance was set at 0.05.
|
Results |
|---|
|
TopAbstractIntroductionSubjects and methodsResultsDiscussionReferences |
|---|
In the study period, there were 2154 operations for aortocoronary bypass and/or valvular surgery. 117 operations were excluded for the following reasons: concurrent performance of other procedures (93), re-operations within 30 days of previous surgery (two), patients on renal replacement therapy (four) or pre-operative serum creatinine >265 mmol/l (3 mg/dl) (18). In all, 2037 operations were included in the study. Among these, seven patients had had two operations and were considered to be at risk of ARF for both.
Thirty-eight of the operations were complicated by ARF, giving a risk of 1.9% (95% CI 1.3–2.6%). For patients with coronary bypass only, the risk was 1.2% (95% CI 0.8–1.9%), and for patients with valvular surgery with or without coronary bypass 4.9% (95% CI 2.9–7.7%). The 30-day mortality for patients with ARF was 45% (95% CI 29–61%), for patients without ARF 2.6% (95% CI 1.9–3.3%).
Of the patients without ARF, 640 were randomly drawn as controls. A comparison between the baseline characteristics of patients with and without ARF according to type of surgery is presented in Table 1
.
|
A multivariate logistic regression analysis of ARF with the same independent variables as in the final logistic model of the CICSP study was performed (n=678). The following risk factors were associated with a higher risk of ARF (P<0.05): valvular surgery, left-ventricular ejection fraction <0.35, the presence of pulmonary rates and chronic obstructive pulmonary disease. From Table 2
, it can be seen that the other risk factors found in the CICSP study were not statistically significant in our analysis. This was to be expected, as the present study had insufficient statistical power to identify all the risk factors. In Table 2, the OR found in the CICSP study are listed for comparison. With one exception (pulmonary rales), the CICSP ORs fell within the 95% CI of the ORs in the present study. The Hosmer–Lemeshow test demonstrated good calibration of the model (
2=8.40, df=8, P=0.40). When the analysis was repeated without patients with missing data, the results were not appreciably different.
|
The points used for calculating the CICSP additive severity score are listed in Table 2
. Table 3 shows its distribution. The score's sensitivity and specificity for predicting ARF were estimated for cut-off levels corresponding to each value of the score and plotted in a ROC curve shown in Figure 1. The area under the curve, which is a measure of the score's discriminatory power, was 0.71, compared with 0.76 in the CICSP study and 0.72 for the QMMI cohort [3,4].
|
|
In Table 3
, the positive and negative predictive values of the score have been calculated for all possible cut-off levels. A cut-off level of 6, i.e. taking a score of 6 or higher to predict ARF, represents a reasonable compromise between sensitivity (0.53) and specificity (0.71). The positive predictive value for this cut-off was 0.03 and the negative predictive value 0.99. The maximum positive predictive value for any cut-off was 0.14.
|
Discussion |
|---|
|
TopAbstractIntroductionSubjects and methodsResultsDiscussionReferences |
|---|
The risk of ARF after cardiac surgery has been the subject of numerous studies. Most of them have tried to assess pre-operative risk factors, but have included too few patients to be able to do this with sufficient statistical power. So far, the only study with adequate power has been the CICSP study, which included more than 43 000 patients [3]. Even so, cross-validation is a crucial step in the development of algorithms for clinical prediction. In the present study, the CICSP additive severity score algorithm was validated in a patient population from a different geographical region and time period than in the original study. Other important differences were a higher percentage of female patients and a lower percentage of patients with diabetes and previous cerebrovascular disease in our cohort. The fact that the algorithm performed well in spite of these differences is an indication of its robustness. The data for the cross-validation were collected retrospectively partly through medical record review. Although this design has the limitation that the model's performance may be favourably biased by differential retrieval of data, an analysis excluding patients with missing data gave no indication that this was the case.
The algorithm has previously been validated in the QMMI cohort [4]. However, this cohort did not include valvular surgery. In the present study, 17% of the patients had undergone valvular surgery with or without surgery on the coronary arteries. Compared with the CICSP cohort, there was a higher risk of ARF among patients with valvular surgery in our hospital. The logistic regression showed that the OR for this type of surgery was higher (4.1 vs 2.0), although the 95% CI included the CICSP OR (Table 2). Another difference was a higher mortality in the CICSP cohort (for patients with ARF 64 vs 45%; for patients without ARF 4.3 vs 2.6%). These differences were both statistically significant. Since ARF and death are competing risks in this situation, one could speculate that the different risk of ARF for valvular surgery could be explained by a higher mortality for the US patients. Unfortunately, the mortality for valvular surgery was not given for the CICSP cohort [3].
As a part of the cross-validation, we wanted to explore the potential for clinical use of the algorithm. A clinical decision based on the algorithm would have to fix a cut-off level of the additive score for deciding which patients were eligible for some intervention, as e.g. withholding surgery or preventive measures. The probability of developing ARF for patients with a score higher than the cut-off is conventionally referred to as the positive predictive value, and the probability of not developing ARF with a lower score as the negative predictive value [16]. In Table 3, these probabilities have been calculated for all possible cut-off levels of the score. As can be seen, the maximum positive predictive value was 0.14, i.e. if the score was greater than the cut-off it was still far more likely that the patient would not develop ARF than the opposite. On the other hand, the negative predictive value was greater than or equal to 0.98 for all cut-off levels.
These results indicate that the algorithm will not be very useful as a tool for making decisions about individual patients. The reason for this is the very low risk of ARF in both the CICSP- and QMMI-cohorts and in our material. As well as being functions of the sensitivity and specificity of the algorithm, the positive and negative predictive values depend on the risk of ARF. To identify these few patients accurately, an algorithm with almost perfect discriminatory power would be needed, which is rarely seen in this kind of clinical decision-making. On the other hand, the algorithm will be useful for estimating risk in groups of patients, e.g. when comparing results from different hospitals.
We conclude that the CICSP algorithm for predicting post-operative ARF was valid for patients undergoing cardiac surgery at our hospital. The risk factors for ARF in the present study were consistent with the risk factors found in studies with greater statistical power. Although the low risk of ARF precludes the prediction of ARF for individual patients, the CICSP algorithm is useful as a tool for risk stratification in populations. In addition, the identification of risk factors lays the foundation for further research into the pathophysiological mechanisms of the condition and possible preventive measures.
|
Acknowledgments |
|---|
We are grateful to Jie Yang, Department of Clinical Research, and to Åshild Halvorsen, Department of Clinical Chemistry, both of the University Hospital of North Norway, for extracting data from hospital databases.
|
Notes |
|---|
Correspondence and offprint requests to: Bjørn Odvar Eriksen, Section of Nephrology, Department of Medicine, University Hospital of North Norway, 9038 Tromsø, Norway. Email: medboe{at}rito.no
|
References |
|---|
|
TopAbstractIntroductionSubjects and methodsResultsDiscussionReferences |
|---|
- Zanardo G, Michielon P, Paccagnella A et al. Acute renal failure in the patient undergoing cardiac operation. Prevalence, mortality rate, and main risk factors. J Thorac Cardiovasc Surg1994; 107:1489–1495
[Abstract/Free Full Text] - Ostermann ME, Taube D, Morgan CJ, Evans TW. Acute renal failure following cardiopulmonary bypass: a changing picture. Intensive Care Med2000; 26:565–571[CrossRef][Web of Science][Medline]
- Chertow GM, Lazarus JM, Christiansen CL et al. Preoperative renal risk stratification. Circulation1997; 95:878–884
[Abstract/Free Full Text] - Fortescue EB, Bates DW, Chertow GM. Predicting acute renal failure after coronary bypass surgery: cross-validation of two risk-stratification algorithms. Kidney Int2000; 57:2594–2602[CrossRef][Web of Science][Medline]
- Conlon PJ, Stafford-Smith M, White WD et al. Acute renal failure following cardiac surgery. Nephrol Dial Transplant1999; 14:1158–1162
[Abstract/Free Full Text] - Suen WS, Mok CK, Chiu SW et al. Risk factors for development of acute renal failure (ARF) requiring dialysis in patients undergoing cardiac surgery. Angiology1998; 49:789–800[Web of Science][Medline]
- Andersson LG, Ekroth R, Bratteby LE, Hallhagen S, Wesslen O. Acute renal failure after coronary surgery—a study of incidence and risk factors in 2009 consecutive patients. Thorac Cardiovasc Surg1993; 41:237–241[Web of Science][Medline]
- Frost L, Pedersen RS, Lund O, Hansen OK, Hansen HE. Prognosis and risk factors in acute, dialysis-requiring renal failure after open-heart surgery. Scand J Thorac Cardiovasc Surg1991; 25:161–166[Medline]
- Schmitt H, Riehl J, Boseila A et al. Acute renal failure following cardiac surgery: pre- and perioperative clinical features. Contrib Nephrol1991; 93:98–104[Medline]
- Mangano CM, Diamondstone LS, Ramsay JG et al. Renal dysfunction after myocardial revascularization: risk factors, adverse outcomes, and hospital resource utilization. The Multicenter Study of Perioperative Ischemia Research Group. Ann Intern Med1998; 128:194–203
[Abstract/Free Full Text] - Aronson S, Blumenthal R. Perioperative renal dysfunction and cardiovascular anesthesia: concerns and controversies. J Cardiothorac Vasc Anesth1998; 12:567–586[CrossRef][Web of Science][Medline]
- Halstenberg WK, Goormastic M, Paganini EP. Validity of four models for predicting outcome in critically ill acute renal failure patients. Clin Nephrol1997; 47:81–86[Web of Science][Medline]
- Lins RL, Elseviers M, Daelemans R, De Broe ME. Problems in the development, validation and adaptation of prognostic models for acute renal failure. Nephrol Dial Transplant2001; 16:1098–1101
[Free Full Text] - Cockcroft DW, Gault MH. Prediction of creatinine clearance from serum creatinine. Nephron1976; 16:31–41[Web of Science][Medline]
- Hosmer DW, Lemeshow S. Applied Logistic Regression. John Wiley & Sons, New York:1989
- Sackett DL, Haynes RB, Tugwell P. Clinical Epidemiology—A Basic Science for Clinical Medicine. Little, Brown & Co., Boston, MA:1985
- Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology1982; 143:29–36
[Abstract/Free Full Text]
Accepted in revised form: 30. 8.02
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  K. Karkouti, D. N. Wijeysundera, T. M. Yau, J. L. Callum, D. C. Cheng, M. Crowther, J.-Y. Dupuis, S. E. Fremes, B. Kent, C. Laflamme, et al. Acute Kidney Injury After Cardiac Surgery: Focus on Modifiable Risk Factors Circulation, February 3, 2009; 119(4): 495 - 502. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 P. Knapik, P. Rozentryt, P. Nadziakiewicz, L. Polonski, and M. Zembala Retrospective cross-validation of simplified predictive index for renal replacement therapy after cardiac surgery Interactive CardioVascular and Thoracic Surgery, December 1, 2008; 7(6): 1101 - 1106. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 A. Candela-Toha, E. Elias-Martin, V. Abraira, M. T. Tenorio, D. Parise, A. de Pablo, T. Centella, and F. Liano Predicting Acute Renal Failure after Cardiac Surgery: External Validation of Two New Clinical Scores Clin. J. Am. Soc. Nephrol., September 1, 2008; 3(5): 1260 - 1265. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
P. Arora, S. Rajagopalam, R. Ranjan, H. Kolli, M. Singh, R. Venuto, and J. Lohr Preoperative Use of Angiotensin-Converting Enzyme Inhibitors/Angiotensin Receptor Blockers Is Associated with Increased Risk for Acute Kidney Injury after Cardiovascular Surgery Clin. J. Am. Soc. Nephrol., September 1, 2008; 3(5): 1266 - 1273. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 M. H. Rosner, D. Portilla, and M. D. Okusa Analytic Reviews: Cardiac Surgery as a Cause of Acute Kidney Injury: Pathogenesis and Potential Therapies J Intensive Care Med, January 1, 2008; 23(1): 3 - 18. [Abstract] [PDF]
|
|
|
|
|
|
J. R. Brown, R. P. Cochran, B. J. Leavitt, L. J. Dacey, C. S. Ross, T. A. MacKenzie, K. S. Kunzelman, R. S. Kramer, F. Hernandez Jr, R. E. Helm, et al. Multivariable Prediction of Renal Insufficiency Developing After Cardiac Surgery Circulation, September 11, 2007; 116(11_suppl): I-139 - I-143. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 C.-Y. Lin, Y.-C. Chen, F.-C. Tsai, Y.-C. Tian, C.-C. Jenq, J.-T. Fang, and C.-W. Yang RIFLE classification is predictive of short-term prognosis in critically ill patients with acute renal failure supported by extracorporeal membrane oxygenation Nephrol. Dial. Transplant., October 1, 2006; 21(10): 2867 - 2873. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
M. H. Rosner and M. D. Okusa Acute Kidney Injury Associated with Cardiac Surgery Clin. J. Am. Soc. Nephrol., January 1, 2006; 1(1): 19 - 32. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 E. H. Kincaid, D. A. Ashburn, J. R. Hoyle, M. G. Reichert, J. W. Hammon, and N. D. Kon Does the Combination of Aprotinin and Angiotensin-Converting Enzyme Inhibitor Cause Renal Failure After Cardiac Surgery? Ann. Thorac. Surg., October 1, 2005; 80(4): 1388 - 1393. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 K. Karkouti, W.S. Beattie, D.N. Wijeysundera, V. Rao, C. Chan, K.M. Dattilo, G. Djaiani, J. Ivanov, J. Karski, and T.E. David Hemodilution during cardiopulmonary bypass is an independent risk factor for acute renal failure in adult cardiac surgery J. Thorac. Cardiovasc. Surg., February 1, 2005; 129(2): 391 - 400. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 R. L. Mehta Acute Renal Failure and Cardiac Surgery: Marching in Place or Moving Ahead? J. Am. Soc. Nephrol., January 1, 2005; 16(1): 12 - 14. [Full Text] [PDF]
|
|
|
|
 |
|
 P. E. Antunes, D. Prieto, J. F. de Oliveira, and M. J. Antunes Renal dysfunction after myocardial revascularization Eur. J. Cardiothorac. Surg., April 1, 2004; 25(4): 597 - 604. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||