Comparison Of Three Meta-analytic Procedures For The Combined R_equivalent Estimation

1.1 PurposeRosenthal and Rubin (2003) proposed a simple procedure for obtaining an estimate of an effect size requivalent (req) from a P value and the sample size. By reviewing literature, it has been cited by researchers to employ in realistic practice including primary study level and meta-analytic context. However, to date, little has been explored about the utilities of types of meta-analytical procedures affect the accuracy and efficiency of estimating req effect size indicator. Hence, goals and research questions of the current study were set as follow.1.The study aimed to1) study the accuracy of three meta-analytic estimators for meta-analyzing requivalent with the presence of either publication bias or heterogeneity, or both, or neither; a) observed req derived from both t-test and non-parametric Wilcoxon test as a measure of effect size for two-group univariate mean comparisons under various specified data conditions;b) meta-analytic estimators included HSr, HOz, HOr. HSr and HOz represent two major methods for meta-analyzing correlations. HSr procedure which analyzes Pearson r correlations using sample-size weights is advocated by Schmidt, Hunter; whereas HOz approach advocated by Hedge and Oklin's analyses Fisher z correlations using inverse-variance weights. HOr is combined the elements from the above mentioned procedures that is the application of inverse-variance weights for r without the Fisher z transformation;2) indentify effect of different statistical properties of primary data conditions on accuracy of meta-analytic estimators;3) clarify the conditions for better using either meta-analytic procedure over another in term of accuracy.2.Research questions1) How the observed req derivation from different statistical properties of primary data conditions assert influence on accuracy of estimating mean req. Properties of primary data conditions included the violation of either one, or two, or three statistical assumptions:equal variance, independence, normality, would impact the accuracy on req estimate. To what extent these effects are carried over to the phrase of meta-analysis.2) How the inferential test from which the observed req derived would impact the accuracy of mean req estimate.3) How the sample size in the primary study level and meta-analysis study level exert influence on the accuracy of mean req estimate. 4) In the phrase of meta-analysis, apart from the number of study and sample size, two other aspects would be addressed, one was the application of meta-analytic approach to req, anotherfu one was the common issue, namely to what extent heterogeneity and publication bias would influence the validity of results from meta-analysis of req.5) Clarify the conditions in which the use of r form or z form for synthesizing req was beneficial over the other.1.2 MethodsMote Carlo simulation accomplished with SAS/IML version 9.1.3 was set out to investigate and explore the nine potential factors that would make impact on the accuracy of three meta-analytic estimators for mean req with the evaluation criteria of bias, MSE, standardized bias and acceptable standardized bias.1.Nine design factors1)number of study (k) with 4 levels:6,30,60,120;2)sample size (n=nl=n2) with 4 levels:10,30,60,120;3)population distribution (pop) with 2 levels:normal distribution, non-normal distribution;4)variance in primary study (var) with 2 levels:equal variance(1:1), unequal variance(1:10);5)correlation in primary data (corr) with 3 levels:independent (corr=0), non-independent (corr=0.2), independent (corr=0.7);6)inferential test (test) with 2 levels:t-test, Wilcoxon-test;7)population effect size (p) with 4 levels:0,0.1,0.3,0.5; 8)heterogeneity in effect size (H) with 2 levels:homogeneity (τ=0), heterogeneity(τ=0.2);9)publication bias (PB):2 levels, without publication bias, with medium publication bias;For convenience of analysis, combination of publication bias and heterogeneity (PB/H) was set and resulting in 4 levels:PB/H=1 represented condition free of publication bias and heterogeneity, PB/H=2 represented condition with publication bias but without heterogeneity, PB/H=3 represented condition without publication bias and with heterogeneity, PB/H=4 represented with publication bias and heterogeneity.2.Three meta-analytic procedures1)Hunter & Schimdt procedure which incorporates untransformed r (HSr)2)Hedges and Oklin's procedure basing on z-transformed (HOz)3)Hedges and Oklin's procedure basing on r (HOr).3.Evaluation criteria1)Bias, standardized bias, MSE, standardized bias, frequency and proportion of conditions with Stdb50 within levels of each study factors among three estimators2)Acceptable standardized bias (Stdb50) was defined as the standardized bias with range of-50% to 50%.3)Each point estimator's bias and MSE were estimated from its estimates'mean and variance over replications; bias, MSE, Standardized bias and Stdb50 for an estimator req ofρwere defined asBias(req)＝E(req-ρ)＝E(req)-ρ, MSE(req)= E[(req-ρ)2]=[Bias(req)]2+Var(req), Stdb(req)＝100%×(req-ρ)/SEreq where SEreq=(?),SEreq is the standard deviation of the req.1.3 ResultWhen p=0,0.5, no cases with Stdb50 for three estimators appeared. Whenρ=0.1, counts of cases with Stdb50 for HOz and HOr estimators were higher than that for HSr estimator:605 for HOz,607 for HOr,297 for HSr. Whenρ=0.3,66 conditions with Stdb50 for HSr estimator emerged. No cases with Stdb50 were observed in HOz or HOr estimators. The results of bias, MSE, standardized bias and Stdb50 for three estimators were recorded and analyzed under various combinations of statistical condition anchoring atρ=0.1,0.3.1. Under conditionsρ=0.11) Combinations of various statistical conditionsIn perspective of statistical characteristics①independent/non-independent (2 level),②equal variance/unequal variance (3 level),③normal/non-normal (2 level),④inferential t-test/Wilcoxon test in primary study level (2 level),24 combinations of cases were examined totally. They stand for the situations that observed req derived from either t-test or Wilcoxon test in the following statistical conditions:①the ideal statistical assumptions fulfillment,②violating one of three statistical assumptions,③violating two statistical assumptions,④violating three statistical assumptions.2) For HSr estimatora) The most count of Stdb50 fell in the category of k=6; under following conditions less cases with Stdb50 when k=120:Unequal variance (1:10), non-independent (corr=0.2), normal data and observed req derived from Wilcoxon test; unequal variances (1:10), independent (corr=0.0), non-normal data and observed req derived from Wilcoxon test; Equal variances (1:1), non-independent (corr=0.2), non-normally distributed data and observed req derived from Wilcoxon test; unequal variances (1:10), non-independent (corr=0.2), non-normal data and observed req derived from Wilcoxon test.b) When n=10, under all 24 statistical combinations, no cases with Stdb50 were observed. The most count of Stdb50fell in the category of n=100.c) The most count of Stdb50 fell in the category of PB/H=2 or 3, namely the presence of either publication bias or heterogeneity. Fourteen of the twenty-four combinations of statistical assumptions included null count of Stdb50 when PB/H=4, namely the presence of both publication bias and heterogeneity and the largest average standardized bias and MSE appeared in PB/H=4.3) For HOr and HOz estimatorsa) The most count of Stdb50 and the largest average standardized bias fell in the category of k=6 in 18 of 24 combinations of statistical assumptions. The largest average MSE presented in k=6 for all combinations of statistical assumptions.b) The most count of Stdb50 fell in the category of PB/H=3 or PB/H=4, namely the either presence of heterogeneity or presence of both publication bias and heterogeneity. The smallest average standardized bias emerged when PB/H=3, namely the presence of heterogeneity in 19 combinations of statistical assumptions. The largest average MSE appeared when PB/H=4, the presence of both publication bias and heterogeneity in all combinations of statistical assumptions. Under following statistical combinations no cases with Stdb50 appeared:Under equal variances (1:1), independent (corr=0.0), non-normal data and observed req derived from Wilcoxon test; Unequal variance (1:10), non-independent (corr=0.2), normal data and observed req derived from Wilcoxon test; Unequal variances (1:10), independent (corr=0.0), non-normal data and observed req derived from Wilcoxon test.c) The most striking difference among the three estimators appeared when the correlation of primary data with 0.7 which violated the non-independent assumption. There were five such conditions under which the magnitude of average standardized bias in HSr estimator was smaller than that in HOz and HOr estimators in all levels of k; while under other 19 conditions, magnitude of average standardized bias in HSr estimator was larger than that in HSr HOr and HOz estimators in all levels of k.2. Under conditionρ=0.366 conditions with Stdb50 in HSr estimator whenρ=0.3. These cases were at n=10 or with publication bias, exclusively. No cases with Stdb50 were observed in HOz or HOr estimators.1.4 Conclusion1.None of the estimators examined resulted in an adequate estimate of req effect size for all of the scenarios investigated. Whenρ=0.1, three meta-analytic procedures-HSr, HOz and HOr under examined produced most accurate estimates for mean req.2.The accuracy of the meta-analytic estimators for req relied hugely on population effect size. Many sources might be the sources of random variation req effect size: data type, inferential test, population effect size, publication bias, heterogeneity in effect size, which made the accuracy of meta-estimator becomes more complex.3.The efficacy of the estimators investigated was condition specific. It specific to conditions whenρ=0.1 andρ=0.3 with other factors similar to those in present study. Basing on result of present study suggestions on choice of meta-analytic estimators were attempted to be made.1) Whenρ=0.1, in other situations, since from the results of the study the difference on the use of r form or z form for synthesizing req was not notable, and considering the easy use of r from, HOr procedure is advocated.2) Whenρ=0.1, In case of the presence of either publication bias or heterogeneity, or both, HOr procedure is recommended. 3) Whenρ=0.3 and n=10, HSr procedure is recommended.4.In the context of current study, when population effect size is null, medium and larger effect size, the choice for meta-analytical procedure synthesizing requivalent effect size should think twice before the final decision is made. It is advisable for primary study researcher reporting their results with appropriate effect size metric and sufficient information for others to further synthesizing work.