Introduction to Meta-Regression Analysis

What is Meta-Regression Analysis?

 


 

Eff'n Effect

 

MRA is the statistical analysis of previously reported regression results (Stanley and Jarrell, 1989). It seeks to summarize and explain the disparate empirical findings routinely reported in nearly all areas of economics. Over the last 25 years, a thousand meta-analyses have been conducted in economics, with over 100 new ones appeaing each year.

.

Eff'n Fun

 

What have meta-analysts learned?

  • That environmental values may be transferred from MRA to unstudied sites (Rosenberger and Loomis, 2000; Smith and Pattanayak, 2002).
  • Regression model misspecification is, in fact, the principal cause for the observed variation among reported economic findings, confirming the concerns famously expressed by Leamer (1983), Summers (1991), Sala-i-Martin(1997), and others.
  • The empirical literature often contains strong evidence against widely held economic theory and contrary to conventional narrative reviews (Stanley, 2001; Stanley, 2004; Stanley, 2005b; Doucouliagos and Stanley, 2009). Without some objective and systematic method of literature reviewing, conventional narrative reviews can draw any conclusions their authors wish.
  • Publication selection inflation is found in the majority of economic research areas, and its effects are often larger than the magnitude of underlying economic phenomenon being investigated (Doucouliagos and Stanley, 2013).

Introductions to meta-regression analysis can be found in Stanley and Jarrell (1989), Stanley (2001), Stanley (2013), and Stanley and Doucouliagos (2012).

Publication Selection Inflation

"Many other commentators have addressed the issue of publication bias ... All agree that it is a serious problem" (Begg and Berlin, 1988, p.421)

"Are all economic hypotheses false?” de Long and Lang (1992) rhetorically asked. Researchers, reviewers and editors treat ‘statistically significant’ results more favorably; hence, they are more likely to be published. Studies that find relatively small and ‘insignificant’ effects are much less likely to be published, because they may be thought to say little about the phenomenon in question. Publication selection has long been a major issue for meta-analysts. Medical researchers are so concerned about the insidious effects of publication selection (recall the deaths due to side effects of Paxil and Vioxx) that their best journals (e.g., JAMA, New England J. M., Lancet, etc.) now require the prior registration of clinical trials if their results are to be published later (Krakovsky, 2004). Publication selection bias is so strong that we are likely to be better off discarding 90% of the research results (Stanley, Jarrell and Doucouliagos, 2010).

“(P)ublication bias is leading to a new formulation of Gresham’s law —like bad money, bad research drives out good” (Bland,1988, p.450).

Funnel graphs should look like this one, below, for the union-productivity literature, though they seldom do.

 

VSL-fun

 

Union-fun

Many economists have turned their attention to the issue of publication selection and have used meta-regression analysis to identify and correct it.

  • Card, D., Krueger, A.B., 1995. Time-series minimum-wage studies: A meta-analysis. American Economic Review 85, 238-43.
  • Görg, H., Strobl, E. 2001. Multinational companies and productivity spillovers: A meta-analysis. Economic Journal 111, F723-40.
  • Ashenfelter, O., Greenstone, M., 2004. Estimating the value of a statistical life: The importance of omitted variables and publication bias. American Economic Review 94, 454-60.
  • Abreu, M., de Groot, H.L.F., Florax, R.G.M., 2005. A meta-analysis of beta-convergence: The legendary two-percent. Journal of Economic Surveys 19, 389-420.
  • Doucouliagos, C., 2005. Publication bias in the economic freedom and economic growth literature. Journal of Economic Surveys 19, 367-88.
  • Nijkamp, P. and Poot, J. 2005. The last word on the wage curve? Journal of Economic Surveys 19, 421-460.
  • Rose, A.K., Stanley, T.D., 2005. A Meta-Analysis of the effect on common currency on international trade, Journal of Economic Surveys 19, 347-65.
  • Stanley, T.D., 2005. Beyond publication bias, Journal of Economic Surveys 19, 309-45.
  • Doucouliagos, H., Paldam, M., 2006. Aid effectiveness on accumulation: A meta study. Kyklos 59: 227-54.
  • Disdier, AC and K Head 2008. The puzzling persistence of the distance effect on bilateral trade. Review of Economics and Statistics 90: 37-44.
  • Nelson, J.P. and Kennedy, P.E. 2009. The use (and abuse) of meta-analysis in environmental and natural resource economics: an assessment. Environmental and Resource Economics 42: 345-77.
  • Doucouliagos, Hristos and Stanley, T.D. 2009. Publication selection bias in minimum-wage research? A meta-regression analysis,” British Journal of Industrial Relations, 47: 406-28.

No other approach can cleanse the economic literature of the distorting effect of publication selection. Economists have begun to develop MRA methods that might ‘solve’ this fundamental problem of empirical science and to render this bias mostly harmless. 

  • Stanley, T.D., 2005a. Beyond publication bias, Journal of Economic Surveys 19, 309-45.
  • Stanley, T.D., 2008. Meta-regression methods for detecting and estimating empirical effect in the presence of publication selection, Oxford Bulletin of Economics and Statistics 70, 103-127.
  • Stanley, T.D. and Chris Doucouliagos,  2012. Meta-Regression Analysis in Economics and Business. Routledge. 
  • Stanley, T.D. and Chris Doucouliagos, 2013, Meta-regression approximations to reduce publication selection bias, Research Synthesis Methods.

In an era characterized by the rapid expansion of research publications and a flood of empirical findings on any given subject, knowledge and sensible policy action are being drowned. All reviews, whether conventional or meta, are vulnerable to publication selection bias. Without some objective and balanced way to integrate this sea of results, ideology and self-serving deceit will dominate the public discussion of economic research. What we need is some objective and critical methodology to integrate diverse research findings and to reveal the nuggets of ‘truth’ that have settled to the bottom. This is precisely what Meta-Regression Analysis (MRA) can do!

References:

  • Begg, C. B., Berlin, J.A., 1988. Publication bias: A problem in interpreting medical data, Journal of the Royal Statistical Society (Series A) 151, 419-445.
  • Bland, J.M., 1988. Discussion of the paper by Begg and Berlin, Journal of the Royal Statistical Society (Series A) 151, 450-451.
  • De Long, J.B. and Lang, K. 1992. Are all economic hypotheses false? Journal of Political Economy 100:1257-72.
  • Doucouliagos, H. and Stanley, T.D. 2012. Theory competition and selectivity: Are all economic facts greatly exaggerated? Journal of Economic Surveys, forthcoming, 2012, available online.
  • Krakovsky, M., 2004. Register or perish, Scientific American 291(Dec.), 18-20.
  • Leamer, E. E. (1983) Let's take the con out of econometrics. American Economic Review 73, 31-43.
  • Rosenberger, R.S., Loomis, J.B., 2000. Using meta-analysis for benefit transfer: in-sample convergent validity tests of an outdoor recreation database. Water Resources Research 36 (4), 1097–1107.
  • Sala-i-Martin, X., 1997. I just ran 2 million regressions. American Economic Review 87, 178-183.
  • Smith, V.K., Pattanayak, S.K., 2002. Is meta-analysis a Noah's ark for non-market valuation? Environmental Resource Economics 22, 271–296.
  • Stanley, T.D., 2001. Wheat from chaff: Meta-analysis as quantitative literature review, Journal of Economic Perspectives, 15, 131-50.
  • Stanley, T.D., 2004. Does unemployment hysteresis falsify the Natural Rate Hypothesis? A meta-regression analysis," Journal of Economic Surveys, 18 (2004), 1-28.
  • Stanley, T.D., 2005a. Beyond publication bias, Journal of Economic Surveys 19, 309-45.
  • Stanley, T.D., 2005b. Integrating the empirical tests of the Natural Rate Hypothesis: A meta-regression analysis,” Kyklos, 58 (2005), 587-610.
  • Stanley, T.D. and Jarrell, S.B., 1989. Meta-regression analysis: A quantitative method of literature surveys," Journal of Economic Surveys, 3 (1989), 161-170.
  • Stanley, T.D and Doucouliagos, Hristos, 2010. Picture this: A simple graph that reveals much ado about research. Journal of Economic Surveys, 24(2010): 170-91.  
  • Stanley, T.D., Jarrell, S. B. and Hristos Doucouliagos, 2010. Could it be better to discard 90% of the data? A statistical paradox. The American Statistician, 64(2010): 70-77.  
  • Stanley, T.D and Doucouliagos, Hristos, 2012. Meta-Regression Analysis in Economics and Business. Routledge.
  • Stanley, T.D, 2013. Does economics add up? An introduction to meta-regression analysis.” European Journal of Economics and Economic Policy 10: 207-220.  

  • Summers, L.H., 1991. The scientific illusion in empirical macroeconomics. Scandinavian Journal of Economics 93, 129-48.