MAER-Net recommends that all meta-analyses in economics should comply with the following reporting protocols.
Research Questions and Effect Size
- A clear statement of the specific economic theories, hypotheses, or effects studied.
- A precise definition of how effects are measured (the ‘effect size’), accompanied by any relevant formulas.
- An explicit description about how measured effects are comparable, including any methods used to standardize or convert them to a common metric.
Research LiteratureSearching, Compilation and Coding
- A full report of how the research literature was searched. This report should include:
- the exact databases or other sources used;
- the precise combination of keywords employed; and
- the date that the search was completed.
- A full disclosure of the rules for study (or effect size) inclusion/exclusion. It is also useful to provide a list of all studies included and a description of why others were excluded.
- A statement addressing who searched, read, and coded the research literature. Two or more reviewers should code the relevant research.
- A complete list of the information coded for each study or estimate. At a minimum,we recommend that reviewers code:
- the estimated effect size;
- its standard error, when feasible, and the degrees of freedom (or sample size);
- variables that distinguish which type of econometric model, methods and techniques were employed;
- dummy (i.e., 0/1) variables for the omission of theoretically relevant variables in the research study investigated;
- empirical setting (e.g., region, market, industry);
- data types (panel, cross-sectional, time series, . . . );
- year of the data used and/or publication year;
- type of publication (journal, working paper, book chapter, etc.); and
- the primary study, publication and/or dataset from which an observation is drawn.
MRA Modeling Issues
- A table of descriptive statistics of the variables that are coded (means and standard deviations) and graph(s) displaying the effect sizes (e.g., funnel graphs, forest plots, bar charts).
- A fully reported multiple MRA, along with the exact strategy used to simplify it (e.g., general-to-specific, Bayesian).
- An investigation of publication, selection, and misspecification biases. When suspected, these should be controlled for in subsequent MRA models.
- Methods to accommodate heteroscedasticity and within-study dependence.
- Results from MRA model specification tests, robustness checks, or sensitivity analyses.
With one possible exception,MAER-Net has come to a clear consensus about these reporting guidelines. The requirement to have two reviewers code all the relevant research has received the most comment and discussion. As economists, we all are acutely aware of the tradeoff between the improved quality that the second coder will likely add (through catching mistakes and resolving ambiguities) and the increased cost (in weeks of highly skilled professional labour). We understand that the highest standards of scientific rigor demand at least two highly-knowledgeable researchers code the relevant research base. Nonetheless,MAER-Net does not wish to prohibit Ph.D. students and researchers at resource-challenged institutions from employing this important tool to understand their areas of research. To finesse these opposing concerns, the above statement is sufficiently broad to encompass a second reviewer randomly checking a substantial proportion of the research literature if the coding protocol is stated explicitly and justified.