External Validation Of Risk Scores For Predicting Ischemic Stroke Short-Term And Long-Term Mortality In Patients In China

Objectives: Acute ischemic stroke results in substantial morbidity andmortality. In caring for stroke patients, clinicians and investigators often needto estimate the risk of short-term and long-term death. Accurate prediction ofthe mortality of acute stroke is important for several reasons. The effect ofstroke management is not confined to treatments by clinicians, but alsoinfluenced by patients and their families. Their confidence can be greatlyenhanced by offering an accurate prognosis. A reliable prognosis allows betterplanning for supportive care, and allocating resources in a more efficient wayfor patients’ relatives. It may also allow patients to be stratified into differentprognostic groups for clinical trials. However, most previous modelspredicting short-term or long-term mortality after ischemic stroke are neitherexternally validated nor easy to implement. In addition, models based onretrospective data may produce information bias; models involving fewerpatients admitted to intensive care unit are of limited use in unselectedhospital-based cohorts; and some models included prognostic variables whichcould not be immediately and routinely collected at admission so that timelyprediction for initiating acute treatment was precluded and the applicability ofthese models were limited. To allow for an almost immediate and accurateprognosis based on a few simple variables, Eric E. Smith et al recentlydeveloped and internally validated models for predicting in-hospital mortalityusing Get With the Guidelines-Stroke (GWTG-Stroke) Program data. On theother hand, the iScore is a recently developed and validated risk score that can be used to estimate the risk ofshort-and long-term mortality after an acute ischemic stroke usingeasy-to-collect clinical parameters and comorbid conditions. In order todemonstrate the accuracy and utility of these models, the current study aimedto apply them to patients in the data set of China National Stroke Registry(CNSR) which differs from the GWTG-Stroke Program data set and theRegistry of the Canadian Stroke Network (RCSN) in terms of nationality. Onthe basis of the original prognostic models, we will address the followingquestions:(1) Are these prognostic models adequate in this differentpopulation in China?(2) Whether the NIHSS score plays a key role inpredicting in-hospital mortality?Methods: The cohorts from the CNSR data sets were used to assist in thevalidation of these risk scores based on GWTG-Stroke Program data andRCSN data set. The study of CNSR was approved by the central institutionalreview board at Beijing Tiantan Hospital. The registry recruited22216consecutive eligible patients who consented from132participating acute carehospitals which generally had more resources and stroke expertise than thetypical Chinese general hospitals, especially the rural hospitals betweenSeptember2007and August2008. The steering committee also attempted toensure representation from each of the27provinces and four municipalities inMainland China. All patients or their designated relatives were informed aboutstudy participation, and informed written consent was obtained.A standard data collection protocol was developed by the steeringcommittee. Paper-based registry forms (PRF) developed by the expertadvisory panel were used for data collection. Information was collectedthrough a face-to-face interview and from the medical records by trainedresearch coordinators.For the risk score based on the GWTG-Stroke Program data The variables analyzed were defined as follows: presentation duringdaytime regular hours was defined as presentation between7AM and5PMMonday to Friday. Past medical history was defined on the basis ofpreexisting conditions, with the exclusion of conditions that were newlydiagnosed during the hospital stay. Stroke severity was assessed by NIHSSscore. Arterial fibrillation in hospital was defined according to the clinicalmanifestation and the findings on the electrocardiogram during the hospitalstay.Details of the selection of variables for the risk score based onGWTG-Stroke Program data, data sources, and the creation andconceptualization of the risk score have been previously published.Eligibility Criteria: patients who met the following criteria were included:(1) ischemic stroke;(2) not presenting at the emergency department becauseof direct floor admission or because of new acute stroke that occurred duringhospitalisation for another reason; exclusion criteria:(1) hemorrhagic stroke ortransient ischemic attack (TIA);(2) transferring from another acute carehospital; and (3) missing data on discharge destination or gender. Thus, a totalof7015patients from130hospitals represented the final external validationsample. The data extracted from them were used for further analysis. Of note,variables with missing data should be imputed as follows: Missing mode ofarrival to the hospital was imputed to private transport (because ambulancepersonnel should have documented arrival times for patients arriving byambulance); patients missing past medical history information were imputedto have no past medical history; and missing arrival time was imputed to theoff-hours or weekend category (the most common category).The main outcome of interest was in-hospital death.Statistical Analyses: Firstly, the overall characteristics of the externalvalidation sample from the CNSR data set were described. For the sake ofcomparing the baseline characteristics among patients who died in the hospitaland those who survived to discharge, the normality of all continuous variables was checked using the Shapiro-Wilk test. In the case of normality, Studentt-test was used; otherwise, Kruskal-Wallis test was used. The differencesbetween qualitative data were analyzed by χ2test (with Yates correction or byFisher exact test when needed). In addition, age-specified, sex-specified andoverall mortality were analyzed and compared.Secondly, the differences of patient characteristics and overall mortalitybetween original and CNSR data sets were analyzed.Finally, the original model was validated in patients from the CNSR dataset for whom complete data on the predictive variables and outcome wereobtainable. Details of the analytic approach for the creation of the original riskscore have been previously published. Variables in the original modelincluded age, sex, NIHSS, mode of arrival, atrial fibrillation, previous strokeor TIA, coronary artery disease, diabetes mellitus, and history of dyslipidemia.The performance of the original predictive model in the external validationsample was determined by generating C statistic and plot of observed versuspredicted mortality with10deciles of predicted risk. The fit of the model wasassessed using Hosmer-Lemeshow goodness-of-fit χ2test, and the model wasregarded significant at P≥0.05. Pearson correlation coefficient was used tocompare the observed and predicted mortality. The curve of receiver operatingcharacteristic (ROC) was also drawn. The C statistic is equivalent to the areaunder the receiver operating characteristics curve and is equivalent to theprobability that the predicted risk of death is higher for patients who died thanfor patients who survived. A C statistic of1.0indicates perfect prediction,whereas a C statistic of0.5indicates no better than random prediction. Inaddition, the C statistics for the age-specified (median age) and sex-specifiedsubgroups in the external validation sample were also calculated. In order toevaluate the value of NIHSS score in prediction of mortality, the C statisticsfor NIHSS alone and for the model that excluded NIHSS score based on theoriginal model were calculated in the external validation sample. Further, the C statistics were compared between the original model and the excludedNIHSS model.For the iScoreStroke severity was assessed on admission with National Institutes ofHealth Stroke Scale (NIHSS) in CNSR, which could be converted to theCanadian Neurological Scale (CNS): an NIHSS score of14to22equals aCNS score of1to4(severe), an NIHSS score of9to13equals a CNS score of5to7(moderate), an NIHSS score of≤8equals a CNS score of≥8(mild),and an NIHSS score of>22equals a CNS score of0(a score of zero wasassigned to patients in a coma). All ischemic stroke subtypes were included inthe present study. Ischemic stroke subtypes was classified according to theTrial of Org10172in Acute Stroke Treatment (TOAST) criteria.Details of the selection of variables for the iScore, data sources, and thecreation and conceptualization of the iScore have been previously published.Eligibility Criteria: The cohort used in the present study from CNSRincluded patients who were≥18years of age with a primary diagnosis ofacute ischemic stroke. Patients with missing baseline characteristics (age,NIHSS score, glucose on admission, and date last seen “normal” before theindex event; n=593,4.78%) and invalid health card numbers were excluded.Patients with missing follow-up outcome or with transient ischemic attack(TIA) were also excluded. Patients with hemorrhagic strokes were alsoexcluded because they have different underlying stroke mechanisms, riskfactors, and prognosis, compared with those with ischemic stroke. The studyof CNSR was approved by the central institutional review board at BeijingTiantan Hospital.Outcome Measures: The main outcomes of interest were30-day mortalityand1-year mortality.Statistical Analyses: The baseline characteristics between the CNSR andthe RCSN cohorts were compared by the means of the χ2test for thecategorical variables, Student t-test for the means and Kruskal-Wallis test for the medians for continuous variables. When needed, Yates correction orFisher exact test were used for categorical data. Details of the analyticapproach for the creation of the iScore have been previously published.Variables included in the iScore used to predict mortality at30days includedage, sex, CNS, stroke subtype (lacunar, nonlacunar, undetermined),preadmission independence, glucose on admission, and presence of atrialfibrillation, congestive heart failure, cancer, or renal failure (on dialysis). Inaddition to these variables, previous myocardial infarction and current smokerwere additionally included in the iScore predicting mortality at1year. Weused iScore quintiles to divide the CNSR cohort into5risk categories.Logistic regression model was used to assess the performance of the iScorefor predicting30-day mortality and1-year mortality. The modeldiscrimination was assessed by the C statistic. The curve of receiver operatingcharacteristic (ROC) was also drawn. The C statistic is equivalent to the areaunder the receiver operating characteristics curve and is equivalent to theprobability that the predicted risk of death is higher for patients who died thanfor patients who survived. A C statistic of1.0indicates perfect prediction,whereas a C statistic of0.5indicates no better than random prediction.Calibration was assessed using Hosmer-Lemeshow goodness-of-fit χ2test,and the model was regarded significant at P≥0.05. However, becauseHosmer-Lemeshow test is known to be oversensitive to small deviations fromgood fit in large samples, we compared predicted versus observed mortality atthe risk score level using Pearson correlation coefficient. The observed andpredicted mortality rates were plotted as continuous function of the riskscores.All analyses were conducted using SAS statistical software (Version9.2,SAS Institute Inc., Cary, NC).Results:For the risk score based on the GWTG-Stroke Program data There were7015patients in the CNSR data set meeting the specifiedinclusion criteria. Admissions were submitted by130hospitals. In-hospitaldeath occurred in205(2.9%) of7015patients. Median age was68.0years.Patients who died were more likely to be older, to have arrived by ambulance,to be severe with much higher NIHSS score (19versus4), and to have ahistory of atrial fibrillation, previous stroke or TIA or coronary artery disease.However, current smoking and daytime admission were associated with lowermortality.There were many differences between the GWTG-Stroke data set andCNSR data set that reached conventional levels of statistical significance. Inthe CNSR data set, madian age was5years younger than in the original dataset; similarly, in-hospital mortality was much lower (2.9%versus5.19%) andmedian NIHSS score was lower (4versus5). Patients in the CNSR data setwere more likely to be transported by private transport, to have a history ofprevious stroke or TIA, to be current smoker, and to have arrived at daytime.The proportion of most past medical histories in the CNSR data set was lowerthan that in the original data set. Although so many differences were observedin demographic characteristics and risk factors between the two data sets,these2populations were still relatively comparable according to thedistribution of death and most of the variables.Age-specified and sex-specified mortality was further analyzed andcompared in the CNSR data set. Mortality in female was higher than that inmale (3.8%versus2.4%, P=0.0008). Age-specified mortality rose doubly withevery10years increase (<65:1.2%;65-74:2.4%;75-84:5.3%;≥85:10.1%,all: P<0.01).In the external validation sample from the CNSR data set, the C statisticfor the original model (0.867,95%CI=0.839-0.895) was significantly greaterthan the C statistic for the NIHSS excluded model which was based on theoriginal model (0.735,95%CI=0.701-0.770, P<0.001). The C statistic for a model that included NIHSS score alone, without anyother predictors, was also very high (0.847,95%CI=0.816-0.879). There wasno significant difference in prediction between the original model and theNIHSS alone (P=0.370).In addition, the original model showed good discrimination in older (>68years, dichotomized at the median) versus younger (≤68years, C statistic0.693versus0.846) patients and in men versus women (C statistic0.856versus0.870), although the C statistic was slightly lower in older patients.A plot of observed versus predicted mortality for the original model inthe external validation sample, grouped into10deciles of predicted risk,showed excellent calibration. The significance level of the Hosmer-Lemeshowtest was0.674. Overall, there was a very high correlation between observedand expected mortality (Pearson correlation coefficient0.978, P<0.0001),again indicating excellent calibration.For the iScoreAmong22216patients enrolled in CNSR, there were12415ischemicstroke patients who consented for the follow up. After excluding patients withmissing baseline characteristics (n=593,4.78%) and patients with missingfollow-up outcome (n=166,1.34%for30-day follow-up and n=771,6.21%for1-year follow-up), there were11656patients with30-day follow-up and11051patients with1-year follow-up left for final analysis. Compared to thepatients in the RCSN, those Chinese patients were younger (Median:67vs.75;Mean:65.5vs.72.0), more likely to be male (61.8%vs.52.6%) and less likelywith severe stroke (CNS≤4:14.6%vs.19.5%), had a lower mortality rate at30days (5.4%vs.12.2%) and at1year (14.3%vs.22.5%).The distribution of the iScore in both cohorts (cohort with30-dayfollow-up and cohort with1-year follow-up in CNSR) was not normallydistributed with median scores of112(Quartile1to Quartile3:95-140) at30days and94(Quartile1to Quartile3:80-114) at1year. Quintiles of theoriginal iScore were used to divide the cohorts into5risk groups. There was a graded increasing in risk of mortality by quintile of risk score, from1.2%forQuintile1to27.0%for Quintile5for the mortality rate at30days, and from3.4%for Quintile1to57.8%for Quintile5for the mortality rate at1year.In addition, we analyzed the predictive ability of the iScore by comparingthe step-by-step modelings. These risk models adjusting for fewer risk factorsdid not permit discrimination between patients with the same age and strokeseverity but with different clinical characteristics or medical history. Forexample, for a low-risk category (age of70years with a moderate stroke), theadditional presence of hyperglycemia (>135mg/dL)(+15points), male sex,and atrial fibrillation (+10points each) would double the predicted30-daymortality rate from7.09%to13.84%. A more impressive increase is observedin the predicted1-year mortality rate from25.31%(derived from the modelincluding only age of60years and severe stroke) to71.21%by adding dialysis(+40points) and to83.39%by adding male sex (+5points) and dependency(+20points).In the CNSR, the C-statistics were0.825(95%CI:0.807-0.843) and0.822(95%CI:0.810-0.833) for30-day and1-year mortality, respectively.Due to the diminished calibration for1-year mortality (Hosmer-Lemeshowtest, P<0.05) compared30-day mortality (Hosmer-Lemeshow test, P=0.143),we also plotted observed versus predicted mortality in the external validationsamples from CNSR for30-day mortality and1-year mortality. There was ahigh correlation between the observed and expected mortality rates (Pearsoncorrelation coefficient,0.925for30-day and0.998for1-year mortality, bothP<0.0001), indicating excellent calibration.Conclusions: The GWTG-Stroke mortality risk model can readily beapplied to a different independent patient population from the CNSR data setwhich is the first nationwide stroke registry in China. In addition, the NIHSSscore provides substantial incremental information on a patient,s short-termmortality risk and is the strongest predictor of mortality. The iScore can be also applied reliably to the different ischemic patientsfrom the CNSR data set. The iScore could be a very useful clinical tool topredict30-day and1-year prognosis, help stroke care decision-making andguide discharge planning. It is recommended to further test the validity ofthese risk models if a larger population-based data set is available.