Resources for Learning and Doing Research Synthesis, Part 1

In this blog I’ll cover diverse topics in meta-analytic methodology.  It isn’t, however, intended as a one-stop resource.  Whether you (plan to) produce or consume research syntheses, teach would-be meta-analysts, or follow this topic for other reasons, you’ll likely benefit from other sources of information and support.  This is especially true if you’re interested in areas of the research synthesis landscape beyond the realm of meta-analytic techniques.

In this two-part post I’ll try to drive visitors away from this blog. 😮  Specifically, I’ll describe several (related) types of resources to consider: organizations and training here in Part 1, and methodological publications, software, and collaborators later in Part 2.

Organizations

During the past couple of decades, organizations devoted to research synthesis have grown in number, size, and scope.  Most of these either conduct research syntheses internally or support the work of affiliated research synthesists.  Some of them additionally offer relevant products and services to the broader research synthesis community, such as educational materials and events (e.g., handbooks, technical reports, training modules, workshops), software, and compilations of resources.  Below are links to four such organizations—the most prominent and active among those I know of—each with a brief comment on how to access most of their methodological resources:

• Cochrane Collaboration: see their website’s “Training” section; see also its “Multimedia” section and their more than 15 Methods Groups (most of which have their own website)
• Campbell Collaboration: see their website’s “Resource Center” section
• Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI-Centre): see their website’s “Learning” and “Methods & Tools” sections
• Centre for Reviews and Dissemination (CRD): see their website’s “Our guidance” and “Information resources” sections

Most of these organizations’ websites are fairly extensive, so finding what’s useful might require investigation and persistence.

Training

Depending on your relevant experience and goals, you might benefit from guided training associated with a course or workshop.  Such opportunities vary widely on several dimensions of their content and format, including their topical scope, mathematical difficulty, length, convenience, and cost.  Below I’ll comment on training specific to research synthesis and that in statistics more generally.

First, some training opportunities focus on meta-analysis or research synthesis.  Most of the aforementioned organizations offer various types of training.  Numerous workshops offered by other groups or individuals range from broad conceptual overviews to intensive training in particular software.  Some universities offer a graduate-level course in research synthesis, often through a social-, behavioral-, or health-science unit (education, psychology, business, biostatistics, epidemiology).

Finding these training opportunities is complicated by their diverse names.  Keywords for searching include “meta-analysis,” “systematic review,” “course,” “workshop,” “syllabus,” and variants thereof.  One useful guide is the 6 March 2011 entry on Alan Reifman’s Intro Stats Page; this lists summer programs in statistics, some of which pertain to meta-analysis.

Second, training in general statistics will likely help you understand and use meta-analysis.  Routine procedures in a typical meta-analysis, such as computing estimates of effect size (ES) and analyzing ESs using simple models, should be comfortable for those with a mastery of basic stats that includes classical/frequentist estimation and inference (e.g., confidence intervals, hypothesis tests) for simple univariate linear models (e.g., ANOVA, regression).  Those interested in meta-analysis for particular types of data, such as with discrete outcomes (e.g., counts, proportions, risks, rates, hazards), would benefit from courses that emphasize those data types (e.g., categorical modeling, survival analysis).

Many meta-analyses, however, involve more complex procedures for challenging situations (e.g., tricky ES extractions, dependent ESs, missing data, publication bias).  It’s unclear what type of standard statistics coursework would help choose and use these techniques responsibly.  One general suggestion is that courses in linear mixed models or their kin (e.g., multilevel or hierarchical models) as well as Bayesian statistics should be valuable, because many meta-analytic models can be handled naturally in these frameworks.  Although these latter topics can be rather mathematically demanding, friendlier treatments are increasingly available.

A final thought on training: If you can’t attend a course or workshop you find appealing, consider requesting the syllabus or materials for self-guided study.  Some instructors or organizations might provide certain materials for free or for a nominal price, especially if you’re affiliated with an academic institution.

That wraps up Part 1.  Tune in for my next post, Part 2, in which I’ll discuss methodological publications (e.g., books, bibliographies), software, and collaborators—including statistical consultants.