The Influence Of Publication Bias On Meta-Analysis Results-A Simulation Study

BackgroundMeta-analysis, as an important statistical tool, is widely used in evidence based medicine, however, its result can be directly influenced if publication bias exists. Publication is generally known as smaller chance of publication for studies with no statistical significance, which will cause system error in combined effect size and the related inferences.Now, a lot of methods have been proposed to detect and adjust for publication bias. For detection, the idea of funnel plot is mostly used, main related quantitative methods include:correlation methods such as Begg’s rank correlation、arcsine transformed rank correlation、Schwarzer rank correlation etc.,regression methods such as Egger regression、Macaskill regression、Peters regression arcsine transformed regression etc.; the fail safe/file drawer method is also widely known, it decides whether publication bias exists by estimating how many non-significant studies are needed to reverse the recent conclusion and whether it’s possible that many studies are not published; Bayesian methods are also increasingly considered by many researchers, such as Copas’s selection model. For correction, main methods are the "trim and fill" method and regression methods which both based on funnel plot, some selection models also provide corrected combined effect size. However, simulations indicate that when there are few studies included for meta-analysis, or when the heterogeneity is quite large, none of the detection methods has satisfied power and type 1 error, correction methods are also with defects. Selection models show some advantages, but due to their complicity and the arbitrary definition of selection function, they only get limited application.Almost all the current methods are concentrating on detection and correction of publication bias, but we still have no idea that generally how much difference publication bias can cause.On the other hand, the detection of publication bias and heterogeneity are mutually influenced. Heterogeneity include statistical heterogeneity and non-statistical heterogeneity, the definition and detection of the latter one are complicated, and with no general standards, so we only consider statistical heterogeneity in this study, that is, the included studies are not from the same distribution. At present, almost all the detection methods only concentrate on the influence of heterogeneity on detection of publication bias, but publication bias can also influence the estimation of heterogeneity, which in fact will decrease the accuracy of combined effect size and the related inferences.ObjectivesThis study will discuss the influence of publication bias on meta-analysis results in different conditions, in combination with its mutual influence of heterogeneity, to lay the foundation of research on detecting and correcting publication bias.MethodsThe study is based on Monte Carlo simulation.Generating original study data:OR is the considered effect size. When simulating the original study data, an average event rate of group A and group B is firstly generated from uni(0.3-0.7), then derive the needed event rates through: Logit(pli)=logit[uni(0.3～0.7)]+log(ORi)/2 Logit(p2i)= logit[uni(0.3-0.7)]-log(ORi)/2 Where ln(ORi)～N(ln(ORT),τ2), τ2 is defined as HG times the average within study variance of the homogeneous meta-analysis dataset when all ORi=ORT. When HG=0 the included studies are homogeneous, when HG>0 the studies are heterogeneous.τ2 is derived from a temporarily generated homogeneous dataset. The sample size of each study comes from a log normal distribution with mean 6 and variance 0.6, group A and group B have the same observations. The number of events of each group comes from a binomial distribution.After an observation of original study data has been generated, it will be decided if it can be published according to its sample size. OR size and P value. If it’s published, it will be included in the biased dataset, and no matter published or not, all the generated data will be included in the unbiased dataset. The program will keep generating data until the number of studies included in the biased dataset reached pre-set number.3000 meta-analysis datasets will be generated under each combination of HG. num and ORT.Estimation methods:3 commonly used methods are considered: Mantel-Haenszel(MH). Inverse-Variance(Ⅳ) and Peto. When estimation method is MH or IV, statistics under both random effects model and fixed effect model will be calculated.Parameter settings:Population OR (ORT):1、1.5、2、2.5、3、3.5、4、4.5、5Number of studies included in the biased dataset (num):3～30, by 3Population heterogeneity (HG):0、1、2、3Evaluation method:By describing the difference of combined OR、variance of combined OR、coverage rate of the 95% CI of combined OR、τ2、I2 between biased and unbiased condition.ResultsThe study describes the influence of publication bias from aspects below:Combined effect size:Publication bias can result in more optimistic OR estimation. When ORT=1 the bias is most severe, about 38% on average, when ORT>2 the bias is quite small (<8%). Heterogeneity can increase the influence of publication bias, when HG=3, ORT=1 bias can reach approximately 51%. If there are few studies included for meta-analysis, the influence of publication bias can be quite unstable. When num=3, the change of OR’s estimation from unbiased to biased datasets is about -0.12 on average, and its SD can reach 0.32. But when the number of included studies increases, the average bias doesn’t show obvious trend. MH method and IV method get a little heavier influence than Peto’s method, but Peto’s method can produce combined OR estimation with larger bias (maybe too conservative), especially when ORT is large. Random effects model get heavier influence than fixed effect model, while the latter produce a little more conservative results. In this study, the influence of publication bias is always fluctuating in all situations. Correction is suggested when true OR<2 or estimated OR<2.15.The variance of combined effect size:Publication bias can increase the variance of combined effect size. When ORt<2, the unbiased dataset will get much more studies than the biased one, so there is a big difference between the sample sizes used to calculate variance, which can cause the difference between biased and unbiased variance. Following research can consider eliminating the influence of sample sizes.The coverage rate of 95%CI of combined effect size:Publication bias can decrease the coverage rate. When ORT=1 the coverage can go down to less than 8%, coverage rate under random effects model declines faster, but it’s still larger than that under fixed effect model. When ORT≥3, coverage of the 95%CI under random effects model can exceed 90% under biased condition. The larger the heterogeneity and/or the bigger the num, the smaller the coverage rate. When num is small, the biased 95%CI is quite wide, so the coverage rate doesn’t change much. MH method and IV method have much the same coverage, but the coverage rate under Peto’s method is smaller. Moreover, when ORT≥2.5, Peto’s coverage under biased condition is larger than that under unbiased condition, this may due to the lower estimation of combined effect size when ORT≥2.5.τ2:Publication bias can decrease τ2. When ORT=1, the decrease can be up to 60%. When ORT is getting larger, the extent of τ2’s decrease is getting smaller. The larger the heterogeneity and/or the more the studies included in biased dataset, the more τ2 will devrease. When ORT=1 figure 3-13 shows no obvious trend, maybe under this condition r2 changes too much that HG and num can’t add more modification. Results estimated using MH and Ⅳ are the almost the same.I2:I2 has a more straightforward meaning than τ2 in the influence of combined effect size. The result shows that publication bias can decrease I2. As ORτ increases, the decrease off I2 becomes milder. When ORT=1,I2 can be 0.43 smaller in average. The existence of heterogeneity can enhance the strength of bias on I2. When num gets bigger, the bias of I2 also shows a mild uptrend. When ORT=1 Figure 3-15 appears an unstable fluctuation. This may because many extreme values occur under this circumstance, especially when HG=0 simultaneously.The relationship between publication bias and heterogeneity:The larger the heterogeneity, the more influence publication bias will cause on combined effect size, and publication bias will also decrease the estimation of heterogeneity in the meantime. So publication bias can be seen as a de-heterogeneity mechanism.ConclusionWhen 1<ORT<2, HG>0, Publication bias can cause an apparent difference in meta-analysis results. Random effects model gets heavier influence than fixed effect model. Different estimation methods show much the same results.The increase of true heterogeneity will increase the influence of publication bias, and publication bias will choose more homogeneous studies resulting in smaller estimation of heterogeneity.