Guidelines for reporting meta-epidemiological methodology research
==================================================================

* Mohammad Hassan Murad
* Zhen Wang

## Abstract

Published research should be reported to evidence users with clarity and transparency that facilitate optimal appraisal and use of evidence and allow replication by other researchers. Guidelines for such reporting are available for several types of studies but not for meta-epidemiological methodology studies. Meta-epidemiological studies adopt a systematic review or meta-analysis approach to examine the impact of certain characteristics of clinical studies on the observed effect and provide empirical evidence for hypothesised associations. The unit of analysis in meta-epidemiological studies is a study, not a patient. The outcomes of meta-epidemiological studies are usually not clinical outcomes. In this guideline, we adapt items from the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) to fit the context of meta-epidemiological studies.

*   epidemiology

Published research should be reported to evidence users with clarity and transparency to facilitate optimal appraisal and use of evidence and allow replication by other researchers. However, for a variety of reasons, the literature is commonly viewed as poorly reported with critical information missing from published reports. This necessitated the development of reporting guidelines. Currently, there are guidelines for reporting randomised trials (CONSORT), observational studies (STROBE), systematic reviews (Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA)), case reports (CARE), qualitative research (SRQR), diagnostic and prognostic studies (STARD), quality improvement studies (SQUIRE), economic evaluations (CHEERS), animal preclinical studies (ARRIVE), study protocols (SPIRIT) and clinical practice guidelines (AGREE). These guidelines are catalogued on the website of the EQUATOR Network (Enhancing the QUAlity and Transparency Of health Research).1 The EQUATOR Network is an international initiative that seeks to improve the reliability and value of health research literature by promoting transparent and accurate reporting and wider use of robust reporting guidelines. Some of these reporting guidelines have extensions for subtypes of research (eg, PRISMA-Equity is used for equity-focused systematic reviews). Some function as a tool for reporting and a tool for assessing the methodological quality (eg, AGREE-II is used to evaluate the rigour and the reporting of clinical practice guidelines).

Meta-epidemiological studies are a fairly new type of study that has been increasingly published in the last two decades,2–4 with no clear reporting standards. Sometimes called methodology research, meta-epidemiological research adopts a systematic review or meta-analysis to provide data for methodological analysis.5 The goal is generally broad but often focuses on examining the impact of certain characteristics of clinical studies on the observed effect, describing the distribution of research evidence in a specific setting, examining heterogeneity and exploring its causes, identifying and describing plausible biases and providing empirical evidence for hypothesised associations. Unlike classic epidemiology, the unit of analysis for meta-epidemiological studies is a study, not a patient. The outcomes of meta-epidemiological studies are usually not clinical outcomes.6–8 

In this guide, we adapt the items used in the PRISMA statement9 for reporting systematic reviews and meta-analysis to fit the setting of meta- epidemiological studies. The rationale for starting with PRISMA is that meta- epidemiological research requires a systematic approach (ie, a systematic review) to select data (the input for methodological analysis). The overarching goal of this guideline is to improve the transparency and clarity of healthcare research.

## Examples of meta-epidemiological studies

Analysis of Cochrane systematic reviews showed that trials with inadequate or unclear allocation concealment yielded larger estimates of effect relative to controlled trials with adequate allocation concealment.10 In a second meta-epidemiological study, non-blinded trials of complementary medicine exaggerated the effect size by an average of 0.56 SD (95% CI –0.71 to –0.41) compared with blinded trials.11 A third example is a study that explored the effect of trials stopped early for benefit on future research.12 In a fourth example, it was shown that in almost 30% of endocrinology studies, the largest treatment effect was observed in the first or second published studies (ie, earlier studies, when studies were ordered chronologically using the date of publication).13 

## Proposed reporting guidelines

We have adapted items of the PRISMA statement to fit a meta-epidemiological study. The items are categorised according to the title, abstract, introduction, methods, results, conclusion and funding and are supported by rationale and examples.

### Title and abstract

Both the title and abstract should identify the report as a meta-epidemiological study, for example, ‘*Early studies reported extreme findings with large variability: a meta-epidemiologic study in the field of endocrinology*’.13 The rationale for this identification is to improve indexing and facilitate retrieval. It also helps clinicians searching for systematic reviews about clinical topics to quickly find relevant evidence and avoid reviewing additional non-clinical meta-epidemiological studies. The abstract should also include a structured summary with subheadings that provide the context, hypothesis, methods, data sources, results, limitations and conclusions of meta-epidemiological studies. Variations of these subheadings are acceptable. The rationale for having this information in the abstract relates to the fact that many readers only read the abstract. These requirements are similar to PRISMA.

### Introduction

The introduction should include the rationale and objectives of the study. This requirement is similar to PRISMA. However, it is more important to explicitly state the hypothesis being tested in meta-epidemiological studies compared with a systematic review summarising evidence about the effectiveness of an intervention (which may or may not have a clear hypothesis and primarily aim at empirically evaluating a clinical outcome).

### Methods

The methods section should specify if the meta-epidemiological study follows a priori established protocol and clarify the eligibility criteria, information sources, search methods, study selection and data collection process, data items collected, summary measures and synthesis methods (including assumptions and computational approximations and methods of additional analyses such as sensitivity, subgroup or meta-regression analyses). These requirements are similar to PRISMA. However, registration of protocols may not be needed, particularly since systematic reviews registration sites are geared more towards clinical questions. Search strategy in a meta-epidemiological study is different because it is not driven by a clinical question. Evaluating and reporting the risk of bias in individual studies may or may not be relevant to the question at hand. The type of bias can also be different. Data extraction is often more complex in meta-epidemiological studies and should be described in details sufficient to allow replication. The methods section of a meta-epidemiological study should describe in details the effect measure used in statistical analysis because it is likely that a clinical audience will not be familiar with such measure (eg, ratio of OR, standardised measures and regression coefficients). Such measures are not commonly used in studies that report on treatment effects (ie, clinical audience would be familiar with OR but not familiar with ratio of OR). Quantifying interagreement between investigators is more critical in a meta-epidemiological study than in a clinical systematic review because of the judgements commonly made to classify data in a meta-epidemiological study. For example, a study evaluated the association between industry affiliation and researchers’ position on the cardiovascular risk of rosiglitazone. Reviewers had to assess how favourable was the researchers’ opinion about rosiglitazone using an ordinal scale.14 This task is fairly subjective and requires judgement. In this case, agreement between reviewers is critical to report and affects the reliability of this meta-epidemiological study (arguably more critical than in a systematic review of clinical outcomes).

### Results

The results section should include the results of the study selection process (how many studies, how many patients, etc). Different from PRISMA, study and patient characteristics need to be tailored to the meta-epidemiological question at hand. Therefore, it may not be important to know patients' age, comorbidities and settings in some questions (although such characteristics may be important in other questions). For example, the meta-epidemiological study showing that trials with inadequate allocation concealment yielded larger estimates of effects (relative to trials with adequate allocation concealment) does not necessarily have to describe patients’ characteristics or comorbidities in each individual trial.10 Publication bias should be evaluated and reported similar to PRISMA recommendations.

### Discussion and conclusions

As with PRISMA, the discussion section should include summary of the main findings and comparison with existing knowledge and should also include the limitations and strengths of the meta-epidemiological study. Limitations at a study level may not be important (as in clinical systematic reviews); however, limitations at the level of the meta-epidemiological study are important. The quality of evidence is a highly important step in a systematic review reporting on a treatment effect15 and is critical for decision making but also may not be relevant to a meta-epidemiological study. Investigators should, however, inform readers of their certainty in the results, even if this was determined narratively16 because formal approaches15 are intended to rate the certainty of evidence about interventions. Lastly, systematic reviews of clinical questions should include in their discussion the implication of their findings on future research and current clinical practice. Meta-epidemiological studies will have profound implications for future research and should report such implications. Meta-epidemiological studies will likely have minimal direct impact on clinical practice. Nevertheless, indirect impact on clinical practice is possible and should be reported when present. For example, a meta-epidemiological study showed that in endocrinology, the largest treatment effect was observed in the first or second published studies.13 Clinically, this should make endocrinologists sceptical about new drugs, laboratory tests or genetic markers and not rush into recommending them to their patients until more studies confirm findings observed in these earlier studies.13 

In table 1, we present PRISMA items with adaptations to fit the setting of meta-epidemiological studies.

View this table:
[Table 1](http://ebm.bmj.com/content/22/4/139/T1)

Table 1 
Proposed items to be used for reporting methodology research, adapted from the PRISMA Checklist ([http://prisma-statement.org/PRISMAStatement/Checklist.aspx](http://prisma-statement.org/PRISMAStatement/Checklist.aspx))

## Discussion and limitations

Reporting guidelines have three goals: to enhance the applicability and use of research results, to allow readers to critically appraise the results and to permit reproducibility and replication of the work, which is essential for accountability and transparency purposes. Meta-epidemiological studies should have similar standards and require reporting guidelines that achieve these three goals.

In this guideline, we propose adaptation to PRISMA items to fit meta-epidemiological studies. Meta-epidemiological studies are inherently heterogeneous in terms of their approach and execution, which is a challenge for current reporting guidelines of systematic reviews (such as PRISMA). For example, a meta-epidemiological study used trial registries as a source of its data.17 Registries are not searched using standard search terms and methods and may not fit the reporting requirement needed for systematic reviews. Entries in trial registries are also dynamic, just like Internet searches. In this case, authors of such study should make every possible effort to describe their work in a reproducible fashion but acknowledge these limitations.17 In addition, reporting guidelines are frequently underused and are not always emphasised by journal editors. A descriptive analysis of published randomised trials in the four highest impact general medical journals shows inconsistencies and non-adherence to the CONSORT statement (eg, allocation concealment and details of blinding were reported only in 12% and 21%, respectively).18 Adherence across all reporting guidelines was estimated to be 86% in another study.19 Evaluation of the instructions given to authors shows that PRISMA guidance was referenced only in 13% of gastroenterology and hepatology journals.20 

Here, we have proposed a reporting guideline to be used when publishing meta-epidemiological research. We hope that these guidelines will be used, modified and enhanced and lead to more clarity and transparency of healthcare research.

## Footnotes

*   Competing interests None declared.

*   Provenance and peer review Not commissioned; externally peer reviewed.

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: [http://creativecommons.org/licenses/by-nc/4.0/](http://creativecommons.org/licenses/by-nc/4.0/) 

## References

1.  Enhancing the QUAlity and transparency of health research. The EQUATOR Network. [https://www.equator-network.org/about-us/](https://www.equator-network.org/about-us/) (accessed 17 May 2017).
    
    

2.   Tsujimoto Y , Tsujimoto H , Kataoka Y , et al . Majority of systematic reviews published in high-impact journals neglected to register the protocols: a meta-epidemiological study. J Clin Epidemiol 2017;84:54–60.[doi:10.1016/j.jclinepi.2017.02.008](http://dx.doi.org/10.1016/j.jclinepi.2017.02.008) 
    
    

3.   Zhang Z , Xu X , Ni H . Small studies may overestimate the effect sizes in critical care meta-analyses: a meta-epidemiological study. Crit Care 2013;17:R2.[doi:10.1186/cc11919](http://dx.doi.org/10.1186/cc11919) 
    
    [CrossRef](http://ebm.bmj.com/lookup/external-ref?access_num=10.1186/cc11919&link_type=DOI) 
    
    [PubMed](http://ebm.bmj.com/lookup/external-ref?access_num=23302257&link_type=MED&atom=%2Febmed%2F22%2F4%2F139.atom) 

4.   Savović J , Jones H , Altman D , et al . Influence of reported study design characteristics on intervention effect estimates from randomised controlled trials: combined analysis of meta-epidemiological studies. Health Technol Assess 2012;16:1–82.[doi:10.3310/hta16350](http://dx.doi.org/10.3310/hta16350) 
    
    [CrossRef](http://ebm.bmj.com/lookup/external-ref?access_num=10.3310/hta16150&link_type=DOI) 
    
    [PubMed](http://ebm.bmj.com/lookup/external-ref?access_num=22935084&link_type=MED&atom=%2Febmed%2F22%2F4%2F139.atom) 

5.   Zhang Z . Meta-epidemiological study: a step by step approach by using R. J Evid Based Med 2016:91–7.[doi:10.1111/jebm.12191](http://dx.doi.org/10.1111/jebm.12191) 
    
    

6.   Zhang W . I-01 META-EPIDEMIOLOGY: BUILDING THE BRIDGE FROM RESEARCH EVIDENCE TO CLINICAL PRACTICE. Osteoarthritis Cartilage 2010;18:S1–S8.[doi:10.1016/S1063-4584(10)00293-1](http://dx.doi.org/10.1016/S1063-4584(10)00293-1) 
    
    [PubMed](http://ebm.bmj.com/lookup/external-ref?access_num=20394830&link_type=MED&atom=%2Febmed%2F22%2F4%2F139.atom) 

7.   Sterne JA , Jüni P , Schulz KF , et al . Statistical methods for assessing the influence of study characteristics on treatment effects in 'meta-epidemiological' research. Stat Med 2002;21:1513–24.[doi:10.1002/sim.1184](http://dx.doi.org/10.1002/sim.1184) 
    
    [CrossRef](http://ebm.bmj.com/lookup/external-ref?access_num=10.1002/sim.1184&link_type=DOI) 
    
    [PubMed](http://ebm.bmj.com/lookup/external-ref?access_num=12111917&link_type=MED&atom=%2Febmed%2F22%2F4%2F139.atom) 
    
    [Web of Science](http://ebm.bmj.com/lookup/external-ref?access_num=000176016900003&link_type=ISI) 

8.   Naylor CD . Meta-analysis and the meta-epidemiology of clinical research. BMJ 1997;315:617–9.[doi:10.1136/bmj.315.7109.617](http://dx.doi.org/10.1136/bmj.315.7109.617) 
    
    [FREE Full Text](http://ebm.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMTUvNzEwOS82MTciO3M6NDoiYXRvbSI7czoyMDoiL2VibWVkLzIyLzQvMTM5LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

9.   Liberati A , Altman DG , Tetzlaff J , et al . The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med 2009;6:e1000100.[doi:10.1371/journal.pmed.1000100](http://dx.doi.org/10.1371/journal.pmed.1000100) 
    
    

10.  Odgaard-Jensen J , Vist GE , Timmer A , et al . Randomisation to protect against selection bias in healthcare trials. Cochrane Database Syst Rev 2011;4:MR000012.[doi:10.1002/14651858.MR000012.pub3](http://dx.doi.org/10.1002/14651858.MR000012.pub3) 
    
    [CrossRef](http://ebm.bmj.com/lookup/external-ref?access_num= 10.1002/14651858.MR000012.pub3&link_type=DOI) 
    
    [PubMed](http://ebm.bmj.com/lookup/external-ref?access_num=21491415&link_type=MED&atom=%2Febmed%2F22%2F4%2F139.atom) 

11.  Hróbjartsson A , Emanuelsson F , Skou Thomsen AS , et al . Bias due to lack of patient blinding in clinical trials. A systematic review of trials randomizing patients to blind and nonblind sub-studies. Int J Epidemiol 2014;43:1272–83.[doi:10.1093/ije/dyu115](http://dx.doi.org/10.1093/ije/dyu115) 
    
    [CrossRef](http://ebm.bmj.com/lookup/external-ref?access_num=10.1093/ije/dyu115&link_type=DOI) 
    
    [PubMed](http://ebm.bmj.com/lookup/external-ref?access_num=24881045&link_type=MED&atom=%2Febmed%2F22%2F4%2F139.atom) 

12.  Murad MH , Guyatt GH , Domecq JP , et al . Randomized trials addressing a similar question are commonly published after a trial stopped early for benefit. J Clin Epidemiol 2017;82:12–19.[doi:10.1016/j.jclinepi.2016.10.006](http://dx.doi.org/10.1016/j.jclinepi.2016.10.006) 
    
    

13.  Wang Z , Alahdab F , Almasri J , et al . Early studies reported extreme findings with large variability: a meta-epidemiologic study in the field of endocrinology. J Clin Epidemiol 2016;72:27–32.[doi:10.1016/j.jclinepi.2015.10.015](http://dx.doi.org/10.1016/j.jclinepi.2015.10.015) 
    
    

14.  Wang AT , McCoy CP , Murad MH , et al . Association between industry affiliation and position on cardiovascular risk with rosiglitazone: cross sectional systematic review. BMJ 2010;340:c1344.[doi:10.1136/bmj.c1344](http://dx.doi.org/10.1136/bmj.c1344) 
    
    [Abstract/FREE Full Text](http://ebm.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE3OiIzNDAvbWFyMThfMS9jMTM0NCI7czo0OiJhdG9tIjtzOjIwOiIvZWJtZWQvMjIvNC8xMzkuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

15.  Murad MH , Montori VM , Ioannidis JP , et al . How to read a systematic review and meta-analysis and apply the results to patient care: users' guides to the medical literature. JAMA 2014;312:171–9.[doi:10.1001/jama.2014.5559](http://dx.doi.org/10.1001/jama.2014.5559) 
    
    [CrossRef](http://ebm.bmj.com/lookup/external-ref?access_num=10.1001/jama.2014.5559&link_type=DOI) 
    
    [PubMed](http://ebm.bmj.com/lookup/external-ref?access_num=25005654&link_type=MED&atom=%2Febmed%2F22%2F4%2F139.atom) 
    
    [Web of Science](http://ebm.bmj.com/lookup/external-ref?access_num=000338511200023&link_type=ISI) 

16.  Murad MH , Mustafa RA , Schünemann HJ , et al . Rating the certainty in evidence in the absence of a single estimate of effect. Evid Based Med 2017;22:85–7.[doi:10.1136/ebmed-2017-110668](http://dx.doi.org/10.1136/ebmed-2017-110668) 
    
    [Abstract/FREE Full Text](http://ebm.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NToiZWJtZWQiO3M6NToicmVzaWQiO3M6NzoiMjIvMy84NSI7czo0OiJhdG9tIjtzOjIwOiIvZWJtZWQvMjIvNC8xMzkuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

17.  Gandhi GY , Murad MH , Fujiyoshi A , et al . Patient-important outcomes in registered diabetes trials. JAMA 2008;299:2543–9.[doi:10.1001/jama.299.21.2543](http://dx.doi.org/10.1001/jama.299.21.2543) 
    
    [CrossRef](http://ebm.bmj.com/lookup/external-ref?access_num=10.1001/jama.299.21.2543&link_type=DOI) 
    
    [PubMed](http://ebm.bmj.com/lookup/external-ref?access_num=18523223&link_type=MED&atom=%2Febmed%2F22%2F4%2F139.atom) 
    
    [Web of Science](http://ebm.bmj.com/lookup/external-ref?access_num=000256361200026&link_type=ISI) 

18.  Ghimire S , Kyung E , Kang W , et al . Assessment of adherence to the CONSORT statement for quality of reports on randomized controlled trial abstracts from four high-impact general medical journals. Trials 2012;13:77.[doi:10.1186/1745-6215-13-77](http://dx.doi.org/10.1186/1745-6215-13-77) 
    
    [CrossRef](http://ebm.bmj.com/lookup/external-ref?access_num=10.1186/1745-6215-13-77&link_type=DOI) 
    
    [PubMed](http://ebm.bmj.com/lookup/external-ref?access_num=22676267&link_type=MED&atom=%2Febmed%2F22%2F4%2F139.atom) 

19.  Samaan Z , Mbuagbaw L , Kosa D , et al . A systematic scoping review of adherence to reporting guidelines in health care literature. J Multidiscip Healthc 2013;6:169–88.[doi:10.2147/JMDH.S43952](http://dx.doi.org/10.2147/JMDH.S43952) 
    
    [PubMed](http://ebm.bmj.com/lookup/external-ref?access_num=23671390&link_type=MED&atom=%2Febmed%2F22%2F4%2F139.atom) 

20.  Panic N , Leoncini E , de Belvis G , et al . Evaluation of the endorsement of the preferred reporting items for systematic reviews and meta-analysis (PRISMA) statement on the quality of published systematic review and meta-analyses. PLoS One 2013;8:e83138.[doi:10.1371/journal.pone.0083138](http://dx.doi.org/10.1371/journal.pone.0083138)