Etymology and usage

The word ‘complex’ derives from an ancient linguistic root, PLEK, which implies weaving, giving words such as plait, pleat, pliant, and plicate. Adding the prefix ‘com’ (Latin cum, meaning ‘with’) to the basic root implies interweaving of several elements, making things complicated, literally woven together. Reflecting this, ‘complex’ is defined in the Oxford English Dictionary as ‘consisting of parts or elements not simply co-ordinated, but some of them involved in various degrees of subordination; complicated, involved, intricate; not easily analysed or disentangled’. No other usages of the word have emerged since it entered English in this form in the 17th century.

Previous definitions

Seeking previous definitions of complex systematic reviews, we did a rapid scope of the literature, with the following results:

• First, we identified a need for distinction between a systematic review of complex interventions and a complex systematic review. Petticrew and colleagues have argued ‘that reviews of complex interventions can themselves be simple or complex, depending on the question to be answered.’8 As outlined below, we argue that a complex systematic review may be designated as such for a variety of reasons, regardless of how complex the unit of analysis is (eg, the intervention(s), procedures, and tests).

• We looked for definitions on the National Institute for Health Research Complex Reviews Unit website, but could not find any, although we did find a list of reasons for the increasing complexity of reviews.9

• Gough and colleagues described ‘multi-component reviews' as a strategy for dealing with complex review questions, but we were still unable to identify an explicit definition of a complex review.10

• We found a definition of a ‘comprehensive review’ in the Joanna Briggs Methods Manual: ‘A systematic review is considered to be a comprehensive systematic review when it includes two or more types of evidence, such as both qualitative and quantitative, in order to address a particular review objective.’11

• We identified an excellent series of articles under the banner ‘Considering Complexity in Systematic Reviews of Interventions’ in The Journal of Clinical Epidemiology, although we were unable to identify an explicit definition of a complex review.12

• We identified a paper by Whitlock and colleagues, who described a complex review as one that ‘evaluate(s) a number of linked clinical questions, multiple interventions or diagnostic tests, different or distinct population groups, and/or many outcomes.'13

Elements of the process

Considering how complex reviews are carried out, we propose that for a systematic review to be designated as ‘complex’, it should demonstrate exceptional aspects in two broad domains: the nature and sources of the data included in the review and the processes of synthesis involved in their interpretation. We have identified eight aspects of complex reviews that underpin these domains, described in the text below and illustrated in figure 1.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Proposed elements of a complex systematic review. 

Several of these aspects reflect the interweaving of disparate elements implied by the origin of the word ‘complex’. We suggest that a complex systematic review should primarily fulfil the following criterion:

• The need for a highly skilled and multidisciplinary team to complete the review. Although the Cochrane Handbook alludes to this,14 we extend this absolute requirement by stating that: the skill mix needed to complete a complex review should include a breadth of content expertise, including end-users of the research, to shape and contextualise the relevance and implications of the review findings. The success of a complex review should depend on highly skilled methodological expertise, without which it would be impossible to complete the review.

This is the only absolute criterion. The other elements (below) are optional, but we suggest that a requirement of any complex systematic review is that any two should be included.

• The use of more than one type of study design (eg, qualitative and quantitative) and more than one type of synthesis approach. Gough and colleagues describe a continuum in synthesis approaches that may, at one end, use a predefined method to aggregate existing data and test a theory, or at the other, use a more iterative synthesis approach and configure existing data to generate a new theory.10 A complex review may do both.

• The inclusion of a large quantity of data. This could include, for example, clinical study reports (CSRs), large and highly detailed documents written for regulators as part of licensing applications for medicinal products.15 However, we are not proposing an arbitrary cut-off for what would constitute a large quantity of data. A systematic review of over 100 published randomised controlled trials might provide as much relevant data as two CSRs, each over 4000 pages long. Context is important. Instead, we propose that researchers should justify whether their review qualifies under this heading and why.

• Assessment of a complete evidence development programme. This cumulative collection of data may be presented chronologically, providing greater insights into the origins and purpose of a technology. For example, combined assessment of preclinical, phase I–IV studies of a technology or evidence of the development of a complex instrument or a large diagnostic tool from the horizon-scanning phase to the postmarketing phase, perhaps starting with a definition of its predicate (ie, the index molecule, instrument, or device from which all other versions and medicinal products have evolved).

• Systematic inclusion of data from several different sources. Most systematic reviews source data from the existing published literature and sometimes from the grey (ie, unpublished) literature as well. However, the validity of a systematic review is threatened by the scope of the data it includes. Problems with publication bias, particularly in relation to pharmaceuticals, are well documented.4 These problems have led to calls for inclusion of data from sources other than just the aggregative summaries in the published literature.2 Including individual participant or patient data may mitigate this to some degree.16 Other sources could include regulatory or registry data, where additional information, for example, on adverse events, may be found.17

• An evaluation of an intervention in relation to the contexts in which it is set, for example, information to describe the process through which the intervention is being implemented or to explain the rationale for the choice of study design, in the light of current knowledge. Both of these types could also include timelines.

• Reviews that simultaneously answer questions from different perspectives and viewpoints: societal, healthcare, funders, patients and carers, healthcare workers, and manufacturers. These could include multiple evaluations and comparisons using different evaluation methods consistent with the different perspectives.

• The use of a particular evidence synthesis method for the first time. A review of evidence synthesis methods used across the fields of nursing, healthcare science and services, and health policy found that many methods were contemporary and probably emerging, implying that more detailed descriptions and validation studies were likely to appear in future.6

Here we provide examples of systematic reviews that can justifiably be termed complex (table 1). All involved a multidisciplinary team.

Thus, based on etymology, usage, previous definitions, and considerations of the elements of which a complex systematic review consists, we propose the following definition:

Complex systematic review n. A systematic review, performed by a multidisciplinary team, consisting of multiple components, large amounts of data from different sources or different perspectives, collectively contributing more than would be expected from their individual contributions, the individual components not being easily coordinated, analysed or disentangled.