Last week I went to a workshop on writing systematic reviews run by SYRCLE. The main focus of this workshop, and indeed the main focus within most of the literature on systematic reviews, is on clinical and preclinical research. However, I think that other disciplines can really benefit from some of the principles of systematic reviewing, so I thought I'd write a quick series on how to improve the scientific rigor of writing reviews within the field of language evolution.
So first thing's first, what is a systematic review? A systematic review is defined (by the Centre for Reviews and Dissemination at the University of York) as “a review of the evidence on a clearly formulated question that uses systematic and explicit methods to identify, select and critically appraise relevant primary research, and to extract and analyse data from the studies that are included in the review.”
This is in contrast to more narrative or literature reviews, more traditionally seen in non-medical disciplines. Reviews within language evolution are usually authored by the main players in the field and are generally on a very broad topic, they use informal, unsystematic and subjective methods to search for, collect and interpret information, which is often summarised with a specific hypothesis in mind, and without critical appraisal, and summarised with an accompanying convenient narrative. Though these narrative reviews are often conducted by people with expert knowledge of their field, it may be the case that this expertise and experience may bias the authors. Narrative reviews are, by definition, arguably not objective in assessing the literature and evidence, and therefore not good science. Some are obviously more guilty than others, and I'll let you come up with some good examples in the comments.
So how does one go about starting a systematic review, either as a stand alone paper or as part of a wider thesis?
Systematic reviews require the following steps:
2. Define in- and exclusion criteria (for literature search)
3. Search systematically for all original papers
4. Select relevant papers
6. Extract data
7. Analyse data (with a meta-analysis if possible)
8. Interpret and present data
In the coming weeks I will write posts on how to phrase the research question of your review, tips on searching systematically for relevant studies, how to assess the quality and validity of the papers and studies you wish to cover, and then maybe a post on meta-analysis (though this is going to be difficult with reference to language evolution because of its multidisciplinary nature and diversity within the relevant evidence, I'll have a good think about it)
Undertaking Systematic Reviews of Research on Effectiveness. CRD’s Guidance for those Carrying Out or Commissioning Reviews. CRD Report Number 4 (2nd Edition). NHS Centre forReviews and Dissemination, University of York. March 2001.