Resources for Learning and Doing Research Synthesis, Part 2

In Part 1 of this two-part post I mentioned a few major organizations that provide resources for research synthesis and described numerous training opportunities.  Here, in Part 2, I’ll discuss the plethora of publications on relevant methodology, options for software, and potential benefits of involving collaborators (e.g., colleagues, consultants).

Methodological Publications

Let’s say you want to read the primary literature on a given methodology topic, maybe because you’re interested in state-of-the-art techniques for addressing a given question.  How many methodological items relevant to research synthesis are available?  I estimate that between 5,000 and 30,000 were disseminated by early 2012—depending largely on how we define “methodological,” “items” (e.g., articles, books, chapters, dissertations, conference abstracts, technical reports), “relevant,” and “disseminated.”  Over 95% of these were produced after the mid-1970s; over 50%, since 2000.  Although a good deal of this work proposes new methods, the majority reviews, evaluates, implements, or demonstrates extant methods or makes other pertinent contributions.  Below I’ll mention two types of resources that might help access this vast and rapidly growing literature that’s scattered over numerous diverse disciplines: books and bibliographies.

First, a word of caution: Not everything published is good, and not everything good is published!  Some peer-reviewed methodological work contains serious errors or poor advice; it’s prudent to check for errata and dissenting opinions.  Some useful methodological advances exist only as grey literature—mostly conference abstracts or technical reports.

More than 70 books dedicated to research synthesis methodology have been published since the early 1980s, including second or later editions of a few volumes.  Many emphasize meta-analysis, while others cover the broader realm of research synthesis in depth.  A few focus on specific statistical software.  Recommending particular books is difficult, because what’s best depends largely on one’s background and aims.  That said, in my opinion either edition of the following handbook stands out as a comprehensive, authoritative guide to many topics most research synthesists will encounter, though its 30-ish chapters, 500+ pages, and considerable detail might overwhelm some newcomers:

Cooper, H. M., & Hedges, L. V. (Eds.). (1994). The handbook of research synthesis. New York: Russell Sage Foundation.

Cooper, H., Hedges, L. V., & Valentine, J. C. (Eds.). (2009). The handbook of research synthesis and meta-analysis (2nd ed.). New York: Russell Sage Foundation.

Several individuals or groups have compiled bibliographies on research synthesis methodology.  These collections vary in size, inclusion/exclusion criteria, usability, and other aspects.  Here are links to the four most comprehensive bibliographies I know of:

• Cochrane Methodology Register: Almost certainly the largest active compilation, with nearly 15,000 items in Issue 4 of 2011, this is also a component of the Cochrane Library; it can be searched directly on the web.
• Methods Library from Scientific Resource Center associated with AHRQ’s Effective Health Care Program: This collection currently contains over 4,000 items and can be searched directly on the web.
• Meta-Analysis Unit (University of Murcia): This website’s bibliographic database currently contains nearly 4,000 items and can be searched directly on the web; the website also gives a list of meta-analysis books.
• ‘Article Alerts’ section of Research Synthesis Methods: This regularly published feature consists of a print component with selected recent articles and a more extensive archive component, which together currently contain over 6,000 items.  Items from both components are available online as an Excel file published with each installment as supporting information; this file should permit transferring items to various users’ reference management software and may be searched using Excel utilities.  I’ll probably elaborate on this in a later post, but for now here are the first four installments:

Hafdahl, A. R. (2010). Article Alerts: Introduction and items from 2009, Part I. Research Synthesis Methods, 1, 81-87. doi:10.1002/jrsm.7

Hafdahl, A. R. (2011). Article Alerts: Items from 2009, Part II. Research Synthesis Methods, 1, 319-326. doi:10.1002/jrsm.24

Hafdahl, A. R. (2011). Article Alerts: Items from 2010, Part I. Research Synthesis Methods, 2, 131-138. doi:10.1002/jrsm.43

Hafdahl, A. R. (in press). Article Alerts: Items from 2010, Part II. Research Synthesis Methods. [available online before printed issue] doi:10.1002/jrsm.56

Software

Computer software facilitates most stages of research synthesis—from formulating the problem through disseminating findings—and is vital for many meta-analytic techniques (e.g., analyses, graphics).  Most tasks can be handled by generic software for managing reference citations or coding and analyzing data, such as spreadsheet, database, or statistical packages.  Specialized software for research synthesis and meta-analysis is also available.  Options vary on several dimensions, including functionality, ease of use, availability, and cost.  They range from short scripts/macros or online calculators for specific tasks to fairly comprehensive packages with sophisticated graphical user interfaces.

I won’t recommend particular software here, in part because this depends heavily on individual preference and circumstance.  For example, I mainly use SAS/IML and Excel when meta-analyzing consulting clients’ data—often large projects with numerous complications—or conducting Monte Carlo simulations for my own methodology research; this combination won’t suit most meta-analysts’ needs.  Moreover, because I write most of my programs myself and rarely use specialized software, I’m ill-equipped to offer recommendations.

I might address software in a future post.  For now, however, I’ll offer four thoughts about identifying useful computing tools:

• Peruse options: For descriptions and reviews of software, see the bibliographies above (e.g., search for “software” or “computer”) or the list of reviews and programs from the University of Murcia’s Meta-Analysis Unit
• Ask colleagues: Confer with colleagues who’ve conducted research syntheses similar to what you’re planning in terms of anticipated data and analyses.  What worked well, and what didn’t?  What would they do differently next time?
• Test-drive it: Before committing to software, obtain a trial version and conduct a small pilot version of your anticipated procedures, using either actual studies from your project or other (real or fictitious) studies.
• Verify it: Don’t assume the software makes correct decisions (e.g., as default options) or yields correct results; where feasible, compare its results with alternative computations (e.g., using other software), perhaps based on worked examples in articles or the software documentation.

Collaborators

Conducting a research synthesis well can be daunting even for a seasoned, statistically savvy researcher who’s an expert in his or her substantive area.  Important issues arise that are unlikely to be encountered in other research endeavors.  Methodology for research synthesis is an academic specialty in its own right.   In this final section I’ll discuss how you might work with colleagues and statistical consultants to reduce your burden while building expertise.

Before leading your own research synthesis, consider collaborating on one with a colleague who’s already conducted at least one.  Try to be involved in or privy to as much of the process as feasible, including higher-level management decisions as well as the menial work primary investigators often delegate.  This active apprenticeship role is an opportunity to see how key steps are carried out—including frequent challenges and messiness—while not yet having to make decisions that are best informed by experience.

Here again, a word of caution: Some experienced research synthesists might be poor role models. Reviews of published research syntheses tend to find that many were conducted or reported poorly.  It’s prudent to compare your collaborative experiences with guidance obtained from other resources described above and in Part 1.

As you learn more and lead your own research syntheses, you might also benefit from inviting collaborators with relevant expertise to help at various stages, such as brainstorming during problem formulation, accessing the “invisible college” while finding studies, and assessing inter-coder reliability.

Finally, you might benefit from involving a statistical consultant who has experience with meta-analysis and other aspects of research synthesis.  I make this “has experience …” qualification because an otherwise capable consultant who hasn’t worked in this domain is less likely to anticipate and responsibly manage issues unique to meta-analysis.  An experienced stats consultant can help with several aspects of planning, conducting, and reporting a research synthesis.

Finding stats consultants is easy, but I’d recommend choosing carefully—especially if s/he’ll charge for services.  Before paying a stats consultant consider seeking out either free alternatives, such as through a consulting center on campus, or a statistically capable colleague who’ll collaborate in exchange for co-authorship.  If you hire a paid consultant, I’d recommend asking colleagues for referrals.  Also, on my consulting web site I link to guidance from the American Statistical Association about hiring a stats consultant (e.g., what to expect, ethics, fees).  Here I’ll just I’ll mention four suggestions based on my 15+ years of consulting experience:

• Plan ahead: Involve the consultant as soon as feasible, because decisions at early stages can influence options and results in later stages.  Also, contacting the consultant well before important deadlines (e.g., at least a few weeks) allows time to make consulting arrangements, and it permits a busy consultant to rotate your project into his or her workload.
• Comparison shop: Consider arranging initial meetings with at least two different consultants, and request their credentials and proposals for comparison.  As in many hiring and purchasing decisions, I’d avoid the temptation to choose based solely on (apparent) cost; consider reliability, compatibility, and other factors.  If more than one candidate seems acceptable, consider hiring one as your primary consultant and another for independent advice on difficult issues.
• Clarify expectations: Be clear—preferably in writing—about what you expect from the consultant and vice versa, especially regarding services to be performed, completion times, and payment.  Bear in mind that predicting completion times accurately for research syntheses is often quite difficult.  Also, be wary if a consultant requests full payment before completing your project.
• Weigh benefits and costs: Choosing to involve a stats consultant can be tricky, especially your first time.  Hiring a good consultant can improve your project’s quality and might help reduce time and other resources (vs. hiring no consultant or a bad one).  These desiderata can conflict, however, when doing certain things better requires more resources.  Consider the following quote—attributed by Howard Wainer to Moses Maimonodes—when deciding whether anticipated benefits of hiring a consultant outweigh expected costs: “If you think doing it right is expensive, try doing it wrong.”

That’s all for my brief survey of resources for learning and doing research syntheses.  I’d appreciate suggestions for other resources, even (especially?) if you read this months or years after I’ve posted it.  If there’s sufficient interest in any aspect of what I’ve described and it can’t be addressed adequately via comments, I’ll consider elaborating in a future post.