Bibliography

As a FREE service to the global research-synthesis community, I actively compile, organize, and disseminate a large bibliography on methodology for research synthesis (Meth4ReSyn).  I’ve done this since early 2002. Before I elaborate on this resource—largely for potential donors who’d like more info—here are quick ways to access it (enjoy!) and support my ongoing effort (thanks!):

• New and improving: Version 0.1 of the Meth4ReSyn library in CiteULike contains more than 10% of the items from my publicly available bibliography.  It’s markedly user-friendlier than its older spreadsheet counterpart, and for many items it has more metadata (e.g., tags/keywords, abstracts, DOIs or other online links).  This blog post about it includes tips for starting.
• Crowdfunding: Building this bibliography consumes hundreds of person-hours annually.  Donations to support this effort fund specific bibliography enhancement tasks (BETs), such as adding items and improving users’ experience.  To help fund the active BET—increasing the Meth4ReSyn library to over 3,000 items (more on this below)—please click the ‘Donate’ button below and contribute an amount you think is appropriate (e.g., $10, $20, $50, $100, $200 in USD); this is processed by PayPal.  Thanks for pitching in!

Progress: 6 donations totaling $420 (7.5% of $5600 goal) from 6 donors (as of 16 July 2013)—see donors below

Note: If you encounter problems donating online or would prefer another payment method, please contact me by electronic mail (hafdahla_gmail~com [replace ‘_’ with ‘@’ and ‘~’ with ‘.’]).

• Older and larger: My complete publicly available bibliography resides in Excel spreadsheets distributed with the ‘Article Alerts’ feature section of of Research Synthesis Methods (RSM).  This collection’s most recent installment contains 7,247 items and is free online with this article (click its ‘Supporting Information’ tab to download the Excel file):

Hafdahl, A. R. (2012). Article Alerts: Items from 2011, Part I. Research Synthesis Methods, 3, 325-331. doi:10.1002/jrsm.1069

Tens of thousands of people worldwide develop, assess, teach, learn, use, or otherwise engage with methods for research synthesis.  Please share this page with anyone who might be interested—colleagues, collaborators, consultants, students, instructors, lab personnel, supervisors, etc.

Overview: What? Why? For Whom? How?

This bibliography consists of usefully organized citations and metadata for many articles and other types of scholarly work relevant to methodology for research synthesis.  In short, it’s a large collection of items on a given topic and a convenient way to access subsets of them.  I’d like it to be this planet’s best bibliography in this academic area!

The ultimate goal motivating my work on this bibliography is to promote better production and consumption of research syntheses.  To that end, this resource facilitates access to the copious, diverse, widely dispersed, and burgeoning methodological literature.  Its primary aim is to help users more quickly find more items that are more relevant, compared to other search strategies—as illustrated by the next section’s specific example.  These advantages depend on which items are included, what metadata is associated with these items, and how users interact with the compilation.

Among the numerous people who can benefit directly from this bibliography are the following:

• (bio)statisticians, other methodologists, and their ilk (e.g., software developers, statistical consultants)
• instructors of and students in courses, workshops, and other training environments
• applied researchers who conduct research syntheses or use results from them
• practitioners and policymakers in evidence-based domains

For some of these users the bibliography might also serve secondary aims, such as helping them get acquainted with core terms and concepts, identify new topics of interest, and find connections among related topics (e.g., validity generalization and benefit transfer, network meta-analysis and meta-analytic structural equation modeling).

Compiling a good bibliography and making it accessible to a variety of users can be challenging.  Considerable time, expertise, and other resources are needed to find relevant items (e.g., search various sources, assess relevance) and process them (e.g., verify or add metadata, document properties and status).  This work is never done: Several hundred relevant items are disseminated each year, further searching for work from past years produces more relevant items, and numerous aspects of users’ experience can be enhanced.

Since starting this project in early 2002, I’ve grown more efficient at most major tasks.  As my bibliography has progressed through major stages of development, my vision for it has evolved and solidified.  By comparing it with other compilations, I’ve realized that in a few years it could feasibly be the largest and most accessible bibliography on this broad topic.  Bringing this vision to fruition will require more time and effort than I’ve contributed to date, which in turn requires financial support from the research-synthesis community and interested others.

The next section’s example illustrates benefits of a bibliography dedicated to research-synthesis methodology.  In subsequent sections of this page I comment on the main motivation for this project, summarize its early stages and my vision and plans for substantial improvements, describe the project’s main time demands, and explain the vital role of crowdfunding in bringing this resource to the research-synthesis community.

Illustrative Topic: Study-Level Variables

A well-designed bibliography dedicated to research-synthesis methodology facilitates efficient searching, largely by increasing recall and precision.  For example, suppose we’d like to find methodological work on study-level variables that may relate to effect sizes.  Popular names for such variables are ‘moderators’ and ‘covariates,’ and considerable methodological work has addressed various aspects of their roles in meta-analyses.

As I envision it, the Meth4ReSyn library will permit us to rapidly (a) find most existing methodological work on study-level variables in meta-analysis and (b) identify a subset of these items that match our specific interests in this topic.  We could accomplish both of these tasks using the library’s controlled vocabulary (e.g., tags/keywords, thesaurus), search utilities, and other features (e.g., abstracts, authors, sorting).  We might also discover and exploit terms for related concepts (e.g., mixed-effects models, meta-regression, subgroup analysis, heterogeneity, benefit transfer, ecological bias, regression dilution bias).

Now suppose we instead used a generic reference database that’s either multidisciplinary (e.g., Google Scholar, Scopus, Web of Science) or fairly discipline-specific (e.g., Current Index to Statistics, ERIC, Medline, PsycINFO).  Simple searches, such as for ‘moderator’ or its variants, will probably suffer from at least two problems:

• Low recall: We’ll miss many relevant items, such as those that don’t use our search term(s) or aren’t in the database.
• Low precision: We’ll find numerous irrelevant items, such as substantive research syntheses or primary studies that examined a moderator, methodological work on moderation in primary studies, or work related to non-statistical meanings of ‘moderate.’

Using more search terms or complex queries (e.g., Boolean expressions) or searching in more databases might reduce one problem but probably not both.  In short, if we’re interested in methodological work on a topic in research synthesis, then searches in the Meth4ReSyn library will probably perform better than similar searches in generic reference databases.  That’s not to say the latter are useless; indeed, some of them could be valuable for finding specific types of work (e.g., citation networks, very recent publications, grey literature).

Motivation: An Embarrassment of Riches

In this section I comment on a vital reason to invest time, effort, money, and other resources in building and maintaining a well-designed bibliography on methodology for research synthesis.  In short, the rapidly growing volume and diversity of methodological work may overwhelm those who seek it.  Similar to the proliferation of substantive studies that stimulated advances in and applications of research synthesis, the body of methodological work in research synthesis has become prohibitively large and complex for many who wish to improve, disseminate, and employ best practices.

As of 2010, well over 5,000 research syntheses are published in English annually; it’s hard to estimate how many are published in other languages or not published.  Most of those substantive applications involve several collaborators (usually between 2 and 9 co-authors but sometimes 10, 20, or more), of whom some are responsible for decisions and activities that hinge on methodological matters.  As noted in a previous section, many others besides producers of research syntheses are interested in research-synthesis methodology (e.g., consumers of research syntheses, instructors and students, methodologists).

Since at least the early 1900s methodologists have been developing statistical and other methods for research synthesis, including various types of systematic review and meta-analysis.  In the late 1970s this work accelerated markedly.  Around 2010 the methodological research in this area was growing by between about 500 and 2,000 items/year, depending on what we consider relevant.  This literature comprises numerous types of contributions (e.g., novel methods or implementations, analytic or empirical evaluations, didactic expositions) disseminated in several forms (e.g., articles, books, chapters, conference papers, dissertations, technical reports).

Growing evidence indicates that many research synthesists’ methods lag behind current best practice.  Even methodologists often neglect to cite advances germane to their contributions.  These oversights are due in part to challenges of keeping abreast of developments in research-synthesis methodology, even within narrow topic domains.  This work is scattered over many sources in several disciplines, and most strategies for accessing it yield only a fraction of what’s relevant (along with much that’s irrelevant).  The best compilations I know of in this area include less than half of the relevant work, seem to systematically neglect work of important types (e.g., statistical methods) or in pertinent disciplines, and use relatively scant metadata.

Simply put, there’s so much work in this area and it’s so diverse—various outlets, numerous disciplines, inconsistent terminology—that finding what one needs can be prohibitively difficult.  The more readily accessible this methodological work is to interested parties, the more efficiently they can develop and use better research-synthesis methodology to advance science, policy, and practice.

History of Parent Bibliography

My systematic work on this bibliography began in January 2002, when I first taught a graduate course on research synthesis at the University of Missouri, though I’d been collecting relevant work haphazardly since the mid-1990s (e.g., while taking my first course in meta-analysis and completing my dissertation).  I disseminated an initial version of this bibliography in late 2004 by announcing it on relevant listservs and providing upon request a set of MS Word files, each containing citations pertinent to 1 of 44 topics.  The following excerpts are from a ‘ReadMe’ file distributed with that initial version:

As of 22 December 2004 the full bibliography included 3,347 potentially relevant articles, dissertations, conference papers, and other works.  Most of these were identified in electronic databases and through reference citations during the summer and fall of 2003.  The main initial findings were from searches of five major electronic databases covering disciplines where research synthesis is popular (Dissertation Abstracts, ERIC, Medline, PsycINFO, ProQuest), … .  These were supplemented by searching the Social Science Citation Index for citations of Glass (1976) and about 20 other heavily cited relevant articles, as well as scanning the reference lists of several relevant books … .

An initial classification scheme comprising 44 topics was developed, based in part on perusal of the 3,347 initial works’ titles.  To date 1,789 (53%) of these works have been classified under one or more of these topics; the remaining 47% will require closer inspection of the abstract or full text to classify.

For the next five years I continued adding items to the bibliography, which I call ReSynRefs, and improving it in other ways (e.g., adding keywords and abstracts, completing and correcting citations), but I didn’t distribute another version until early 2010.

Birth of ‘Article Alerts’ in RSM

In early 2009 the founding co-editors of RSM invited me to edit a feature section based on my bibliography.  A central aim behind this section was to collect relevant methodological work from various disciplines into one readily accessible source for various interested audiences.  This was named ‘Article Alerts’ (RSM-AA) to emphasize its inclusion of recently published articles.  As described in the first installment, this feature section consists of two components:

• Print component: Each published installment of RSM-AA includes a set of articles listed in the journal’s print and online versions.  These articles are selected from highly relevant work published during a recent year, and they’re organized into categories and supplemented with keywords.  The four installments published as of September 2012 have included a total of 400 articles from 2009 and 2010.
• Archive component: Accompanying each published installment is an Excel file provided as online supporting information.  This file contains both the print component and a larger collection of items that’s more inclusive than only highly relevant recent articles—work that’s at least moderately relevant and disseminated in any of a variety of outlets during any year.  As of the fourth installment, this includes 5,618 items in addition to the print component’s 400 articles.

Any item added to RSM-AA is first added to ReSynRefs, so the former is a subset.  As of November 2012 ReSynRefs includes just over 800 items not yet included in RSM-AA: about 600 whose relevance is ambiguous, and a couple hundred that have been challenging to process (e.g., incomplete citation).

Transition to User-Friendlier Meth4ReSyn Library

The limited usability of RSM-AA is a major drawback for that publicly available version: Its spreadsheets’ arrangement as a two-way array isn’t conducive to efficient searching and provides little functionality for activities related to retrieving documents or discovering new topics.  Moreover, updates to that version are made only with each published installment.

As noted in the next section, making my bibliography user-friendlier is a core objective in my vision for its future.  To that end, in July 2012 I began transporting the RSM-AA print and archive components to the Meth4ReSyn library in CiteULike; for many items this entails adding metadata (e.g., tags/keywords, abstracts, DOI names).  Besides permitting immediate access to updates, this library offers many enhancements to users’ experience via CiteULike’s environment, such as the following:

• Free online access doesn’t require registration, though registering for a free account permits additional features.
• Searches can be simple or complex in several ways, such as by limiting to specific fields or using Boolean operators or wildcards.
• A simple controlled vocabulary is implemented using tags (a.k.a. keywords, descriptors), which are hyperlinked for convenience.
• Users may export the entire bibliography or subsets of it in several formats.
• Several social features facilitate interactions among CiteULike users, such as sharing items, establishing connections, joining groups, blogging, and messaging.

This blog post gives quick tips for starting to use the Meth4ReSyn library.  After all RSM-AA items are transported to it, this library will house the publicly available version of my bibliography for the foreseeable future.  Whether I’ll continue to update RSM-AA after that remains to be seen.

Going Forward: Vision and Plans for Improvements

During the past decade I’ve devoted a few thousand (unpaid) hours to compiling my bibliography and improving its value for users.  As of August 2012, however, it probably contained only 25% to 35% of the relevant literature, and its publicly available versions were either much smaller than that (Meth4ReSyn) or much less user-friendly than feasible (RSM-AA).  I’m confident that with adequate support I can do much better, and I’m excited to improve this resource’s utility for the research-synthesis community.

To guide this endeavor, in the summer of 2012 I wrote a vision and specific plans for soon creating this planet’s best database on methodology for research synthesis, used heavily worldwide, with a team helping me and stable funding to support our efforts.  Below I elaborate on key elements of the previous sentence.

• Soon: In about 5 years (before 2018) I’d like this bibliography to be the best in this academic domain, with a well-coordinated team and smoothly running processes for updating and maintaining the online database and for generating funding.  I’ve also planned shorter time frames for specific sub-goals (e.g., transport RSM-AA to Meth4ReSyn by summer 2013, create rich thesaurus by summer 2015).
• Best:This bibliography’s main purpose is to help a variety of users efficiently find most work relevant to numerous topics in research-synthesis methodology.  Besides increasing the recall and precision of users’ searches, it should also facilitate retrieving documents, exploring this knowledge domain, and related activities.  I believe such a resource’s quality depends mainly on how comprehensive, current, and user-friendly it is:
  • comprehensive: contains a very high percentage of relevant items, including both published peer-reviewed articles as well as several other (possibly unpublished) types of work from numerous fields
  • current: contains most relevant work disseminated recently (e.g., before the past year)
  • user-friendly: offers easy access and a convenient interface with features for various types of users (e.g., novice vs. expert, occasional vs. frequent), such as simple and complex searching, exporting citations, a rich controlled vocabulary (e.g., thesaurus/taxonomy, tags/keywords), and ways to find related work
• Used heavily: I’d like to promote this resource actively and widely to the large global community of researchers who develop, disseminate, learn, apply, and otherwise engage with these methods—probably well over 50,000 people worldwide in any given year.
• Team: Enlisting more people could improve the bibliography’s quality, hasten progress, increase efficiency, and reduce the project’s reliance on me.  For instance, advisors could provide input on major decisions (e.g., enhancement priorities), experts could help with technically or otherwise challenging specialized tasks (e.g., information/library scientist, web developer), and assistants could perform easier tasks (e.g., search for, screen, and retrieve items; retrieve or create metadata; enter data; write documentation; monitor quality; assess usability).
• Funding: Given the time, effort, and expertise required to accomplish this project’s goals soon—at least 2,000 person-hours/year for at least 5 years—monetary compensation is vital.  I view this bibliography mainly as a continually improving resource for the research-synthesis community, not as a lucrative for-profit business venture.  Hence, I see financial support as fair compensation for the person-hours required to provide a valuable service.

I’m optimistic that early financial support will stimulate a positive feedback loop linking the above elements: By fueling enhancements to the bibliography it’ll increase usage, which will in turn attract more funding, facilitate enlisting reliable collaborators, and expedite completion.  After commenting on time demands in the next section, in the final section below I elaborate on matters of funding, including remarks about plans for specific enhancements.

Where the Time Goes

Rome wasn’t built in a day, nor will be this bibliography.  Some readers might wonder how this project could take several thousand person-hours over several years.  It’s useful to bear in mind that systematically building a comprehensive, current, and user-friendly bibliography in a diverse domain presents different challenges than, say, haphazardly compiling a selective or bibliography on a relatively narrow topic.  Below are the tasks that consume the bulk of person-hours; many of them rely heavily on subject-matter expertise, which limits the availability of qualified personnel.

• Search: Some effective strategies for comprehensive searching are fairly time-consuming, such as treeing forward or backward in citation networks or browsing specific journals or other sources (e.g., conference programs or proceedings).  Finding very recent or very old work tends to take especially long, as does finding grey or fugitive literature.  Harvesting hits from a search query is quick for many electronic databases, but constructing good search queries takes time.
• Assess relevance: Search hits are assessed for relevance to research-synthesis methodology.  For some items this is quick and accurate based on the title alone, but for many others a confident decision requires skimming the abstract/summary or text proper.  Although this judgment takes less than 30 seconds for most items and less than 5 seconds for some, difficult items might require a couple minutes.  Of the many thousands of items I assess (e.g., about 30,000 disseminated in each recent year), fewer than 10% are at least moderately relevant.
• Assign tags: Most relevant items are assigned one or more tags (a.k.a. keywords) that describe the content. The time spent assigning an item’s tags depends on features of the document (e.g., length, clarity, topical diversity) and whether the assignment is based on the title, abstract/summary, full text, or other information.  Also, developing a controlled collection of tags takes time.
• Add other metadata: Depending on an item’s source, it may need additional metadata not obtained during the search, such as an abstract, identifier (e.g., DOI name, PubMed ID, ISBN), or elements of the reference citation.  This often entails searching for the item online or retrieving its full text.

Non-trivial time is also spent on other tasks, such as de-duplicating hits from multiple searches, checking and correcting reference citations and metadata, and retrieving electronic or paper copies.  Also, developing a better controlled vocabulary—currently an unstructured set of just over 100 tags—will require substantial time going forward; this will probably entail collaborating with a taxonomist, library or information scientist, or other expert to create, test, and maintain a more thorough taxonomy or thesaurus.

On the basis of time documented for major tasks, achieving my vision for this bibliography may require about 2,000 person-hours/year (about 40 hrs/wk) during the next several years.  Although that time might be reduced somewhat by technological advances that facilitate automating certain tasks (e.g., text mining to screen search hits), incorporating those advances also requires time and expertise.  If enough people use this bibliography sufficiently often, however, their increased benefits and decreased costs could far outweigh the time, money, and other resources invested in this project.

Support via Crowdfunding: Many Small Donations

A simple idea underlies the current funding model: Money permits more time to be spent on the bibliography.  I’m sufficiently experienced and motivated to pursue this endeavor, so time and money are its main limiting factors.  I’ll gladly devote 2,000 hours/year to this project if I’m partially compensated for the reduction in my income from statistical consulting.  Also, a key component of my vision involves eventually enlisting advisors, experts/specialists, and assistants, most of whom will be compensated financially for at least some of their time.

One of numerous strategies for funding this bibliography is to charge each user for access, but for several reasons that option seems undesirable.  I prefer to instead request donations tied to improvements.  More specifically, I periodically ask many potential users and other supporters to donate small amounts toward well-defined bibliography enhancement tasks (BETs), such as adding items or upgrading the user interface.  I’m optimistic that the large research-synthesis community is sufficiently generous that their (your?) contributions will adequately support this project’s financial needs.

Active BET

The active BET, for which I began accepting donations on 25 September 2012, entails transporting a random sample of 40% more items from RSM-AA’s 4th installment to the Meth4ReSyn library, so the latter contains more than 3,000 items.

On the basis of transporting the first 10% in about 100 hours—about 10 minutes/item on average—and my improved efficiency and familiarity with CiteULike, this BET is projected to require 280 to 360 person-hours.  Much of this time will involve adding metadata (e.g., abstracts, DOI names, tags) to many items.  I’m requesting $5600 (USD) to fund this BET.  This amounts to a relatively low hourly rate but seems reasonable given that the items are already available in RSM-AA.

To support this active BET by contributing any amount you think is appropriate (e.g., $10, $20, $50, $100, $200 or more in USD), please click the ‘Donate’ button below (processed by PayPal); if you encounter problems donating online or would prefer another payment method, please contact me by electronic mail (hafdahla_gmail~com [replace ‘_’ with ‘@’ and ‘~’ with ‘.’]).  With their permission, contributors of $200 or more will be listed below as major donors; contributors of less, as other donors.   Thanks to all who pitch in!

• Donation goal: $5600 USD
• Progress (as of 16 July 2013): 6 donations totaling $420 (7.5% of goal) from 6 donors
• Major donors, $200 or more (as of 16 July 2013): Biostat Inc.
• Other donors, chronologically (as of 16 July 2013): Carol Woods, Terri Pigott, John Sideris, Mark Lipsey, Andy Field
• Anticipated completion date for BET (contingent on funding and workload): 15 August 2013

Completed BETs

Below are brief descriptions of completed BETs, including completion dates and major donors.

0. Created ReSynRefs and RSM-AA, including fairly comprehensive search for items from 2009.  Completion date: 16 January 2012.  MAJOR DONORS ($200 or more): n/a (completed before initiation of crowdfunding)
1. Created Version 0.1 of Meth4ReSyn library (1st 10% of 6,018 items in 4th RSM-AA installment).  Completion date: 20 August 2012.  MAJOR DONORS ($200 or more): n/a (completed before initiation of crowdfunding)

Future BETs

Below are BETs anticipated for the next five years (i.e., through 2017).  These fall into four groups: transport items from RSM-AA and ReSynRefs (#1, #4), improve user-friendless (#2, #6), add recent work (since 2009; #3, #5, #7, #10, #13, #16), and add older work (before 2009; #8, #9, #11, #12, #14, #15).  I’ve listed these future BETs in approximately the order I plan to accomplish them, but this may change based on evolving priorities or users’ feedback.

1. Transport remaining 50% of items in RSM-AA’s 4th installment to Meth4ReSyn.
2. Develop guidance for users of Meth4ReSyn (e.g., quick-start guide, tutorials, answers to FAQs).
3. Update Meth4ReSyn with results from comprehensive search for items from 2010.
4. Update Meth4ReSyn with relevant ReSynRefs items from before 1994.
5. Update Meth4ReSyn with results from comprehensive search for items from 2011.
6. Develop thorough controlled vocabulary (e.g., taxonomy, thesaurus) with user-friendly interface.
7. Update Meth4ReSyn with results from comprehensive search for items from 2012.
8. Update Meth4ReSyn with results from comprehensive search for items from 2008.
9. Update Meth4ReSyn with results from comprehensive search for items from 2007.
10. Update Meth4ReSyn with results from comprehensive search for items from 2013.
11. Update Meth4ReSyn with results from comprehensive search for items from 2005 and 2006.
12. Update Meth4ReSyn with results from comprehensive search for items from 2002 – 2004.
13. Update Meth4ReSyn with results from comprehensive search for items from 2014.
14. Update Meth4ReSyn with results from comprehensive search for items from 1997 – 2001.
15. Update Meth4ReSyn with results from comprehensive search for items from before 1997.
16. Update Meth4ReSyn with results from comprehensive search for items from 2015.

When this plan is accomplished the Meth4ReSyn library will be markedly user-friendlier and will contain most of the relevant work published through 2015.  During this period other improvements might also be made, such as updating RSM-AA, searching for work on specific topics or disseminated in specific forms (e.g., book chapter, dissertation, conference presentation, software), or incorporating mechanisms for feedback from users (e.g., corrections, contributions of fugitive literature, suggestions for enhancements).

---