Developing A Methodological Framework For Determining Decision Thresholds Of Effect Size On Health Outcomes

Background: Healthcare decisions making should be based on the best evidence,taking full account of the benefits and harms of interventions,certainty of evidence,costs and feasibility.However,it is difficult for decision-makers to assess the health impact of interventions fairly and consistently,because decisions are often made without explicit thresholds.The Decision Threshold of Effect Size(DTES)for health outcomes can help decision-makers classify the effect size of an intervention as no effect,small,moderate or large,but there is no systematic method for determining DTES.Objective: The aim of this study is to develop a scientific,systematic and operational methodological framework for the determination of DTES for health outcomes,thereby promoting scientific,standardized and transparent healthcare decision-making processes and ultimately improving population health.Methods:(1)Initial methodological framework development: The China National Knowledge Infrastructure,Wanfang,China Biomedical Literature Database,VIP,Pub Med,Web of Science and Embase databases were systematically retrieved to identify published studies on the determination of DTES for health outcomes,both domestic and international.The general concepts and steps related to DTES determination in the eligible studies were extracted and the extracted information was analyzed descriptively.We wrote the first draft of the framework,invited methodologists to form a core team,and refined the framework through several rounds of core team discussion.(2)Pilot application of methodological framework: In the context of developing a living,rapid evidence-based practice points for the drug treatment of Coronavirus Disease 2019(COVID-19),using the initial methodological framework as the guidance,we determined the DTES for COVID-19 important outcomes(COVID-DTES),validated the rationale and operability of the framework and refined its steps.We conducted a systematic review of the literature to identify the need for the determination of COVID-DTES.The COVID-DTES working group was formed,and a high quality core outcome set(COS)that met the application scope of COVID-DTES was selected.Important outcomes were identified through Likert 9-level scoring method.Systematic reviews that met the scope were screened as the evidence,corresponding scenarios were constructed in the Delphi consultation questionnaire,and the Delphi method was used to obtain expert opinion and consensus.For dichotomous data,we used the relative risk as measure of effect,and for continuous data we calculated the mean difference.The positive coefficient of expert(questionnaire recovery rate),concentration degree of experts’ opinions(K),coefficient of variation(CV),Kendall’s concordance coefficient(W)and expert authority coefficient(Cr)were calculated.Each outcome corresponded to three COVID-DTES: DTES for small effect(DT1),DTES for moderate effect(DT2),DTES for large effect(DT3).Consensus was reached for each outcome when K ≥ 70%and CV ≤ 0.25.For non-consensus COVID-DTES,a face-to-face expert consensus meeting would be held.The final COVID-DTES of important outcomes were determined by discussion and voting,with consensus being reached when at least 70%of the experts agreed.(3)Finalization of the methodological framework: Based on the experience of pilot application,we gathered expert feedback to revise the framework,refine its description and improve the operating points.Using purposive sampling and the principle of information saturation,clinical or public health experts with expertise in health research methodology were invited to participate in a semi-structured in-depth interview on how to optimize the methodological framework.The interview material was coded and analyzed according to grounded theory,and experts’ comments on the framework were collated and summarized in three steps to further refine the framework and develop a revised draft: open coding,axial coding and selective coding.A representative expert consensus group was formed to reach consensus on the framework using the Delphi method,considering factors such as geographical distribution,professional fields and gender.Expert positive coefficient and Cr were calculated to indicate the reliability and authority of the experts involved.To reach a consensus,the agreement rate among the experts should be at least 70%.Steps or operating points that did not reach consensus would be revised according to the experts’ opinion and included in the next consensus round.Formal suggestions for optimizing the framework were collected,and the final framework was established after expert review.Results:(1)Development of initial methodological framework: The search identified seven studies.According to study objective,three eligible studies focused on determining the DTES for specific populations and outcomes,and four eligible studies provided concepts and considerations for DTES determination.We drafted the initial methodological framework based on synthesizing existing research included in the SR.Five methodological experts,each with 5-15 years of research experience,a Ph D degree and a senior title,were invited to form the core team.After three core team discussions to revise and improve the draft,an initial framework of eight steps and associated operating points was developed.(2)Pilot application of initial methodological framework: The COVID-DTES working group consisted of a steering committee,a secretariat group,an evidence review group and a consensus expert group.A total of 15 COVID-19 frontline clinical experts,epidemiologists and methodologists were invited to the consensus expert group.After systematic search and screening,five eligible COSs were obtained,and one high quality COS was selected to form the initial outcomes list.Using the 9-point Likert scale,six important outcomes were identified according to target population subgroups.Eight scenarios were constructed,a Delphi consultation questionnaire was designed,and a round of Delphi expert consensus was conducted.There was consensus on COVID-DTES for all outcomes except for length of hospital stay,with K ranging from73.33% to 100%.In a round of face-to-face expert consensus meeting,consensus was reached on the COVID-DTES for length of hospital stay,and the experts ultimately confirmed and verified all COVID-DTES.For COVID-DTES of mild and moderate patient group: rate of viral clearance closest to 7 days(DT1=1.18,DT2=1.48,DT3=1.74),time to symptom resolution(DT1=1.63 days,DT2=3.01 days,DT3=5.37days);for COVID-DTES of severe and critical severe patient group: mortality(DT1=0.88,DT2=0.73,DT3=0.56),rate of viral clearance closest to 7 days(DT1=1.19,DT2=1.40,DT3=1.63),rate of mechanical ventilation(DT1=0.83,DT2=0.67,DT3=0.47),duration of mechanical ventilation(DT1=1.72 days,DT2=3.74 days,DT3=6.25 days),length of hospital stay(DT1=2.35 days,DT2=4.54 days,DT3=7.83days)and time to symptom resolution(DT1=2.34 days,DT2=5.49 days,DT3=9.23days).During the pilot application,we refined the operating points of steps and confirmed the operability of the initial framework.(3)Finalization of methodological framework: Based on the discussions of members of pilot application working group,three main suggestions were summarized and adopted,relating to terms explanation,pre-consultation training and evidence selection of scenarios.Six health research methodologists with multidisciplinary backgrounds were invited to participate in the semi-structured in-depth interview.The experts were all Ph Ds with 10.83 ± 4.17 years of professional experience.The interview data were coded in three steps and the 62 initial concepts were distilled and condensed into 11 categories,then integrated and summarized into six main categories: setting up expert consensus group,designing DTES questionnaire,selecting survey methods,collecting and analyzing data,convening expert consensus meeting and overall framework proposals.The experts’ suggestions were summarized according to the main categories and the framework was refined to develop a revised draft.Fourteen experts were invited to participate in a round of Delphi consensus,50% male and 50% female,with an average of 12.29 ± 5.21 years of experience.The questionnaire recovery rate was 100%,the Cr was 0.874 > 0.7,and the percentage of agreement for all consensus items ranged from 78.57% to 100%.We received 30 open-ended expert comments from the Delphi consensus.Finally,an 8-step methodological framework for determining DTES was developed:(1)assessing the need for determining DTES,(2)forming a DTES working group,(3)selecting important outcomes,(4)constructing scenarios based on best evidence,(5)designing a DTES expert consultation questionnaire,(6)conducting expert consensus and analyzing the results,(7)convening expert consensus meeting,and(8)dissemination and validity evaluation.The final draft of the framework was explained in terms of the operating points of each step,sent to the experts for review and confirmation,and then the final framework was unanimously approved.Conclusion: The scientific and operational methodological framework for the development of DTES of health outcomes,culminating in the development of 8 steps and associated operating points for determining the DTES of health outcomes,provides a reliable tool for establishing evaluation criteria to reasonably measure the effect size of interventions.Specific measures will be taken to promote its dissemination,application and improvement in the future,and further research will be conducted on multiple approaches for determining DTES,evaluating the validity of DTES,and comprehensively considering the perspectives of target population in determination process.