Examples of studies comparing different sources of data for the same trials
| Reference | Type of study | Intervention comparisons | Source comparison | Take home message |
| Chan, et al 32 | Cohort study of 102 RCTs registered with scientific-ethical committees in Denmark, 1994–1995. | 75% drug trials, 12% counselling/lifestyle trials, 11% surgery/procedure, 2% equipment | Protocols vs publications | ‘62% of trials had at least one primary outcome that was changed, introduced or omitted’. In 40 of 82 trials, prespecified primary outcomes were not presented as such in the journal publication. In 11 trials, outcomes not prespecified were reported as the ‘primary outcome’ in the publication. ‘The reporting of trial outcomes is frequently incomplete and biased and inconsistent with protocols’. |
| Turner, et al 33 | Review of 74 RCTs for 12 antidepressants reviewed by the FDA, and their corresponding publication (or lack thereof) in the literature | 12 antidepressants vs placebo | Medical officer reviews vs publications | Non-publication and selective reporting occurred frequently, and can change the apparent risk-benefit assessment of drugs. Publicly available medical officer reports are a valuable source of unbiased information about clinical trial design and results. |
| Eyding, et al 26 | Systematic review of 13 trials. 76% of patient data unpublished: 86% (1946 of 2256 patients) for reboxetine vs placebo and 67% (1760 of 2641 patients) for reboxetine vs SSRIs | Reboxetine for depression vs placebo or vs other SSRIs included in IQWIG HTA report | CSRs vs publications | The addition of unpublished data changed the direction and conclusions of the efficacy and harms analyses. Published data vs full dataset overestimate benefits by 99%–115% vs placebo and 19%–23% vs other SSRIs. |
| Jefferson, et al 34 | Cochrane review of 25 trials (15 oseltamivir, 60% unpublished, those published had been ghostwritten and corresponding ‘authors’ had no access to study data) | Neuraminidase inhibitors for influenza vs placebo | CSRs vs publications | Lack of detail in publication and unexplained discrepancies when compared with CSRs led the authors to change methods compared with previous version of the review and include only regulatory data, significantly changing the conclusions of the review. |
| Coyne, et al 35 | Review of the NHT run in the 1990s on 1265 haemodialysis patients with cardiac disease | Epoetin lower (9–11 g/dL) vs higher (13–15 g/dL) doses to increase haematocrit to reduce mortality and improve survival and QoL | CSR vs publication | ’Disclosure of these (CSR) results in the 1998 publication or access to the FDA filed report on the NHT in the late 1990s would likely have led to earlier concerns about epoetin safety and greater doubts about its benefits'. |
| Wieseler, et al 15 | Systematic review of 29 studies included in 16 HTA reports prepared by IQWIG during 2006–2011 | 16 different pharmaceuticals mainly for depression and type I and II diabetes | CSRs vs publications vs register entries | CSR consistently reported more information than registers or journal publications. |
| Wieseler, et al 36 | Systematic review of 101 trials with full CSR available included in 16 HTA reports prepared by IQWIG. The study population is the same as Wieseler 2012 but in this study the authors quantified information gain for patient-relevant outcomes graded from 1 to 4 | 16 different pharmaceuticals mainly for depression, asthma and type I and II diabetes | CSRs vs publications vs register entries (unclear which trials have been registered where. Also some trials were conducted in the late 1980s) | CSRs reported complete information on 78%–100% of benefit outcomes vs 20%–53% in combined publicly available sources. The authors estimated 13% publication bias. CSRs reported complete information on 84%–92% of harm outcomes vs 27% to 72% of combined publicly available sources. 15% NR by publicly available sources for both general harms and withdrawals due to possible harms. |
| Rodgers, et al 24 and Fu, et al 25 | Systematic review of 13 trials and 4 single-arm studies (10 and 1 journal published) | Recombinant human bone morphogenetic protein 2 for spinal fusion vs iliac crest bone graft | IPD vs CSRs vs journal publications | Wealth of extra detail from CSRs provided by manufacturer. Fu et al conclude that ‘Early journal publications misrepresented the effectiveness and harms through selective reporting, duplicate publication and under-reporting’. |
| Doshi and Jefferson9 | Descriptive review of 78 CSRs | 14 different pharmaceuticals and biologics | CSRs vs publications (comparison in size) | The ratio of CSR pages to publication pages for available full CSRs with a corresponding publication (‘compression factor’) ranged from 379 to 8805. |
| Vedula et al 31 | Review of transparency and accuracy of reporting of the numbers of participants, description of types of analyses and criteria for including participants in the analysis in 11 published trials | Gabapentin vs placebo for four off-label uses (migraine prophylaxis, treatment of bipolar disorders, neuropathic pain and nociceptive pain) | CSRs accessed from litigation with their published counterparts (21 trials identified, 11 assessed, 8 trials excluded because unpublished, 1 not randomised, 1 no CSR available) | Probably biggest discrepancies occurred between protocol and publication. The authors conclude “we found that the trial publication was not a transparent, or accurate (presuming that the research report truly describes the facts), record for the numbers of participants randomized and analyzed for efficacy". |
| Maund, et al 10 | Review of nine trials in 1999–2001 (seven journal published) | Duloxetine vs placebo | CSR vs publications vs register entries; 1/9 R1 and 9/9 R2 | 7 S published 2 NS unpublished 1 NS published as S after post hoc analysis not mentioned in the paper. Harms 50% and 25% participant reporting inconsistency in two trials, one death in active arm in unpublished trial; lack of clarity on phase of deaths Suicide NR<2% in register reports. SAE three articles failed to report, register entries unclear. |
| Le Noury, et al 29 | RIAT publication, restoring Glaxo SmithKline’s trial 329 run in the 1990s and journal published in 2001 | Paroxetine vs placebo and imipramine vs placebo | IPD with CRFs for 34% (93/275) participants and CSR vs publication | Paroxetine was reported as safe and effective in company-sponsored ghost written publications. Access to CSR data led the authors to conclude that the drug was no more effective than placebo and was toxic in adolescents. The authors identified four outcomes cited in the protocol but not reported in the CSR and publication. |
| Köhler, et al 37 | Systematic review of 15 dossier assessments by AMNOG submitted to IQWIG between 2011 and 2015. The authors assessed completeness of reporting in each document category | 15 different drugs including anti-HIV and oncology | AMNOG documents: IQWiG dossier assessments and publicly available modules of company dossiers vs non-AMNOG documents: EPARs vs journal publications vs register entries available at market entry date point | ’At the time of market entry of a new drug, a substantial amount of information needed for assessment of the corresponding clinical studies and for understanding of the drug’s benefits and harms is missing in publicly available European public assessment reports, journal publications and registry reports (non-AMNOG documents)’. |
| Beaumier, et al 38 | Cochrane review update of 4 CSR (three journals published in four publications) | Olanzepine vs placebo | CSRs vs publications | Dilution due to different coding of similar events (eg, ‘nervousness’, ‘anxiety’ and ‘agitation’). Long-term harms not reported in publications. One suicide in active arm NR in publication; one death in active arm from CV causes identified from FDA drug-approval package not reported in either CSR or publication. Two suicide attempts not reported in active arm in publication and S dose-response with metabolic syndrome NR in a journal publication. |
| Cosgrove, et al 39 | Review of data considered by regulators for registration vs other data available to them vs publications and comparison of regulatory vs SR process | Vortioxetine vs placebo (four RCTs) or active comparator (six studies) for depression | FDA drug-approval package (based on 10 short-term RCTs) and EMA EPAR (12 RCTs) vs publications. At least three studies were unpublished (38% of randomised participants). All unpublished studies showed no difference with comparator | ’Published literature gives the impression that vortioxetine is efficacious, safe and well tolerated, when in fact the data were not collected or analysed in a way that provides sound empirical support for this conclusion’. The authors note extensive sponsor ties of 8/10 authors of published studies and comment on regulatory practice which focuses on an in-depth analysis of ‘positive’ trials rather than the whole evidence base. |
| Hodkinson et al 30 | Exploratory review to assess the reporting of harms in Orlistat trials | Orlistat vs placebo | 5 Roche CSRs vs five journal publications | Journal publications provided insufficient information on harms outcomes compared with CSRs. Serious adverse events, were not reported or mentioned in the journal publications. Overall, CSRs provide extensive information about harms for study methods, including design, conduct, and analysis of the trial. |
| Jureidini et al 40 | Litigation documents vs publication | Citolapram vs placebo | Comparison of 750 documents from the Celexa and Lexapro Marketing and Sales Practices Litigation and publication | ’The published article contained efficacy and safety data inconsistent with the protocol criteria. Procedural deviations went unreported imparting statistical significance to the primary outcome, and an implausible effect size was claimed; positive post hoc measures were introduced and negative secondary outcomes were not reported; and adverse events were misleadingly analysed. Manuscript drafts were prepared by company employees and outside ghostwriters with academic researchers solicited as ‘authors’. |
| Schroll et al 41 | Descriptive review of seven RCTs to assess the reporting of AEs | Orlistat vs placebo | 7 CSRs from Roche vs protocols vs journal publications | ’Study identified important disparities in the reporting of adverse events between protocols, clinical study reports and published papers. Reports of the trials systematically understated adverse events. Based on the study findings, systematic reviews of drugs might be improved by including protocols and CSRs in addition to published articles’. |
| Mayo-Wilson, et al 42 | Impact assessment to determine whether disagreements among multiple data sources of the same trials affected meta-analytic effect estimates, statistical significance and interpretation | Gabapentin and quetiapine | 21 gabapentin RCTs (74 reports, 6 IPDs) and 7 quetiapine RCTs (50 reports, 1 IPD) | ‘Disagreements across data sources affect the effect size, statistical significance and interpretation of trials and meta-analyses’. |
AMNOG, Arzneimittelmarktneuordnungsgesetz (Germany’s Act on reform of the market for medicinal products); CRF, case report forms; CSR, clinical study reports; CV, cardiovascular; EMA, European Medicines Agency; FDA, Food and Drug Administration; IQWIG, Institute for Quality and Efficiency in Healthcare, Germany; NA, not applicable; NHT, Normal Hematocrit Trial; NK, not known; NR, not reported (by the authors); NS, statistically not significantly different; QoL, quality of life; R1, registration 1 (in public register); R2, registration 2 (in manufacturer register); S, statistically significantly different; SAE, serious adverse events;
SSRI, Selective serotonin reuptake inhibitors; IPD, Individual Patient Data; EPAR, European public assessment reports .