You are here

Article Text

Article menu

Download PDFPDF

Original research
Mental Health
Blinding in randomised clinical trials of psychological interventions: a retrospective study of published trial reports
Free
  1. Sophie Juul1,2,
  2. Christian Gluud2,
  3. Sebastian Simonsen1,
  4. Frederik Weischer Frandsen1,
  5. Irving Kirsch3,
  6. Janus Christian Jakobsen2,4
  1. 1 Stolpegaard Psychotherapy Centre, Mental Health Services in the Capital Region of Denmark, Gentofte, Denmark
  2. 2 Copenhagen Trial Unit, Centre for Clinical Intervention Research, Copenhagen University Hospital, Copenhagen Ø, Denmark
  3. 3 Program in Placebo Studies, Harvard Medical School, Boston, Massachusetts, USA
  4. 4 Department of Regional Health Research, The Faculty of Health Sciences, University of Southern Denmark, Odense, Denmark
  1. Correspondence to Sophie Juul, Stolpegaard Psychotherapy Centre, Mental health services in the Capital Region of Denmark, Gentofte 2820, Denmark; sophie.juul{at}regionh.dk

Abstract

Objectives To study the extent of blinding in randomised clinical trials of psychological interventions and the interpretative considerations if randomised clinical trials are not blinded.

Design Retrospective study of trial reports published in six high impact factor journals within the field of psychiatry in 2017 and 2018.

Setting Trial reports published in World Psychiatry, JAMA Psychiatry, Lancet Psychiatry, American Journal of Psychiatry, British Journal of Psychiatry, or Psychotherapy and Psychosomatics.

Main outcome measures Blinding status of participants, treatment providers, outcome assessors, data managers, the data safety and monitoring committee, statisticians and conclusion makers, if trialists rejected the null hypothesis on the primary outcome measure, and if trialists discussed the potential bias risk from lack of blinding in the published trial report.

Results 63 randomised clinical trials of psychological interventions were identified. None (0%; 95% CI 0% to 5.75%) of the trials reported blinding of all possible key persons. 37 (58.7%; 95% CI 46.42% to 70.04%) trials reported blinding of outcome assessors. Two (3.2%; 95% CI 0.87% to 10.86%) trials reported blinding of participants. Two (3.2%; 95% CI 0.87% to 10.86%) trials reported blinding of data managers. Three (4.8%; 95% CI 1.63% to 13.09%) trials reported blinding of statisticians. None of the trials reported blinding of treatment providers, the data safety and monitoring committee, and conclusion makers. 45 (71.4%; 95% CI 59.30% to 81.10%) trials rejected the null hypothesis on the primary outcome(s). 13 (20.7%; 95% CI 12.48% to 32.17%) trials discussed the potential bias risk from lack of blinding in the published trial report.

Conclusions Blinding of key persons involved in randomised clinical trials of psychological interventions is rarely sufficiently documented. The possible interpretative limitations are only rarely considered. There is a need of randomised clinical trials of psychological interventions with documented blinding attempts of all possible key persons.

  • psychology
  • medical
  • methods
  • psychiatry

Data availability statement

All data relevant to the study are included in the article or uploaded as supplementary information.

https://doi.org/10.1136/bmjebm-2020-111407

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Summary box

    What is already known about this subject?

    • Participants, treatment providers, outcome assessors, data managers, the data safety and monitoring committee, statisticians and conclusion makers must ideally be blinded to protect against bias in any trial. The extent of blinding of all possible key persons has not been studied in randomised clinical trials of psychological interventions.

    What are the new findings?

    • Blinding of key persons is rarely reported or implemented in randomised clinical trials of psychological intervention, and only few trialists consider the bias risk this may have caused. Therefore, there is a risk that previous randomised clinical trials of psychological interventions generally may have overestimated the beneficial effects and underestimated the harmful effects of the experimental interventions being studied.

    How might it impact clinical practice in the forseeable future?

    • Randomised clinical trials of psychological interventions with attempts of blinding of key persons are needed and can help to identify effective psychological interventions at low risk of harm.

    Introduction

    Blinding in randomised clinical trials refers to the methodological principle of preventing bias by withholding information about allocation status from individuals or groups.1 Several studies have demonstrated that lack of blinding is responsible for biased treatment estimates.2–12 For example, meta-analyses of trials with and without reports of blinding of participants and outcome assessors have indicated an overestimation of the beneficial effects of the studied interventions in trials without blinding.7–9 These results were confirmed in a metaepidemiological study of 228 Cochrane meta-analyses, in which exaggerated effect estimates were found in trials without blinded participants, personnel and outcome assessors.10 However, a new metaepidemiological study investigating 142 Cochrane reviews with both blinded and non-blinded trials on any topic was recently published without showing effects of blinding.13 Both metaepidemiological studies included, but were not limited to, psychosocial and behavioural interventions. The latter study conflicts with the previously described studies, as it indicated no evidence of a difference in estimated treatment effect between trials with and without blinded participants, healthcare providers and outcome assessors.13 14 Nevertheless, this study may have been at risk of type II errors as it only included approximately half the number of meta-analyses compared with the metaepidemiological study by Savović et al.10 Furthermore, the authors concluded that risk of bias due to lack of blinding should be carefully considered.13 For a detailed comparison of the two metaepidemiological studies, see online supplemental file 1. Based on previous evidence, blinding should remain a methodological safeguard in randomised clinical trials.13–15 It may be argued that blinding of key persons in a trial does not affect the bias risk of ‘hard’ outcomes, such as mortality. However, a large metaepidemiological study did not observe consistently different bias for subjective outcomes compared with mortality.10 If the method of measurement of the hard outcome is insufficiently performed and described, blinding may theoretically matter. For example, if mortality is not systematically assessed, for example, by looking in all participants’ medical journals or person registries, if these are available, then the outcome may be biased if outcome assessors are not blinded. Therefore, trialists should continue to make every effort to incorporate possible blinding procedures into their trial designs regardless of the type of outcome, and readers of trial reports should look for reports of which key persons were blinded to avoid potentially biased conclusions.16 17

    Supplemental material

    Trial participants, treatment providers, outcome assessors, data managers, the data safety and monitoring committee, statisticians and conclusion makers should ideally all be blinded to minimise the risk of biased trial results regardless of the type of studied interventions.6 18–20

    In box 1, we describe how all these key persons may be blinded in randomised clinical trials of psychological interventions.

    Box 1

    How to blind key persons in randomised clinical trials of psychological interventions

    Blinding of trial participants and treatment providers (performance bias)

    Trial participants and treatment providers are more difficult to blind in randomised clinical trials of psychological interventions compared with pharmacological trials, which typically achieve blinding with matching placebo pills.95 106 However, there is evidence that most participants and treatment providers in pharmacological trials break blind.107 108 Despite the obvious challenges of implementing blinding procedures for participants and treatment providers, it is theoretically and perhaps also practically possible to blind both participants and treatment providers in randomised clinical trials of psychological interventions. Motivated potential therapists who have no or limited knowledge about psychotherapy could be trained and supervised in psychotherapy techniques without knowing the exact type of psychotherapy, and participants with no prior experience with psychotherapy may be kept blinded throughout the trial if they are not informed about the specific therapy (the name of the psychotherapy tradition, etc). However, it is plausible that blinding of participants and treatment providers will compromise the effects of the active ingredients of the psychological intervention. Effective delivery of a particular psychological intervention may require extensive training, which would be difficult to implement with the blinding intact.

    Blinding of outcome assessors (detection bias)

    In randomised clinical trials of psychological interventions, implementing efficient blinding procedures for outcome assessors is relatively simple; non-blinded participants may be instructed to withhold information about allocation status when assessed by an independent assessor.18 Efforts to avoid unblinding furthermore include locating the outcome assessors separately from treatment providers, and to assign new outcome assessors in cases of unintentional unblinding.

    Whenever psychological trials deal with effects on participant-reported outcomes (eg, symptoms, mood, quality of life), the interpretation of potential intervention effects becomes marred with lack of blinding, because the participant, who is often unblinded, is the outcome assessor.

    Blinding of data managers

    Blinding of data managers can be achieved by having blinded research personnel handle data entry, data coding and data cleaning. When data are collected in paper participant report forms (PRFs), transferring of data from paper to the electronic database should be performed by blinded research personnel, and data should then be validated by double entry. When using electronic PRFs, blinded data entry can be achieved by having blinded outcome assessors enter data directly into the electronic database, for example, while interviewing the participant. Once data collection is complete, the coding and cleaning of data should be performed with intervention groups concealed from the data manager.

    Blinding of the data safety and monitoring committee

    The data safety and monitoring committee should receive a blinded data set on a prespecified time point during data collection and based on the results of the interim analyses decide if the trial should stop or carry on. The data safety and monitoring committee may request unblinded data throughout the trial period, but the initial data analyses should be blinded to protect against bias in decision-making.

    Blinding of statisticians

    Bias may also be introduced during the statistical analysis of the trial results through the selective use and reporting of statistical tests.109 This may be a conscious or unconscious process spurred by trialists or statisticians eager to see a certain result.16 The best method to avoid this potential bias is first to publish a detailed statistical analysis plan before the analyses begin along with a protocol predefining the trial methodology in detail. Second, the statistician should be kept blinded until the entire statistical analysis has been completed.110 The statistician should receive data from the data manager with the intervention groups concealed as, for example, ‘A’ and ‘B’ and should perform the predefined analyses while the blinding is intact.

    Blinding of conclusion makers

    Blinding of conclusion makers can be done by having the trial steering committee receiving blinded statistical analyses from the blinded statistician with interventions coded as, for example, A and B. The steering committee should then write and agree on two abstracts: one based on the assumption that the experimental intervention is ‘A’, and another based on the assumption that the experimental intervention is ‘B’. Once the steering committee has written and agreed on the two abstracts, the blinding may be broken.110 111 This procedure could reduce the likelihood of post-hoc rationalisation of results in any trial not limited to psychological interventions.

    Because of the difficulties of blinding participants and treatment providers in randomised clinical trials of psychological interventions, it is important to assess the extent of blinding of these and other more easily blinded key trials persons in the current literature. Further, the extent of blinding of key persons involved in a randomised clinical trial of psychological interventions has not yet been systematically studied. The objective of the present study is to evaluate the extent of blinding in trials and the interpretative considerations if the trials are not blinded in six high impact factor journals.

    Methods

    Search strategy

    Two investigators (SJ, FWF) independently searched for randomised clinical trials of psychological interventions (eg, psychodynamic therapy, cognitive–behavioural therapy, or mentalisation-based therapy) for any type of psychiatric disorder published in 2017 or 2018 on the websites of six high-journal impact factor psychiatric journals:

    We chose these peer-reviewed journals due to their high impact factor (ranging from 7.2 to 34.0 in Web of Science for the year 2018). The time period was chosen randomly to provide an overview of the current practise. The two investigators (SJ, FWF) independently screened titles and abstracts for the two selected publication years. Full texts were retrieved for all trial reports if these were judged to be eligible, or if further information were needed to assess eligibility. Any discrepancies were resolved through discussion or, if required, through discussion with a third investigator (JCJ).

    Eligibility criteria

    We included any randomised clinical trial (as defined by trialists) assessing the effects of psychological interventions for any psychiatric disorder, that is, we included (1) trials comparing two or more psychological interventions with each other, (2) trials comparing psychological interventions with non-psychological interventions (eg, drugs, no intervention, or wait-list), and (3) trials assessing the effects of a psychological intervention without a treatment provider (eg, virtual reality exposure therapy with virtual therapists or computerised cognitive–behavioural therapy). In the latter group of trials, the domain ‘blinding of treatment providers’ was classified as ‘not applicable’.

    Data extraction

    Two independent investigators (SJ, FWF) extracted data and performed risk-of-bias assessments. Any disagreements were resolved through discussion or, if required, they consulted a third investigator (JCJ). For each trial, we extracted if the following key persons were described as blinded to treatment allocation: (1) participants; (2) treatment providers; (3) outcome assessors (for the primary outcome); (4) data managers; (5) the data safety and monitoring committee; (6) statisticians; and (7) conclusion makers. We also extracted if trialists rejected the null hypothesis when reporting results on their primary outcome(s) at maximum follow-up, and if trialists discussed the potential bias risk from lack of blinding of the psychological intervention in the published report. Both the published trial reports and any online supplemental materials were used for data extraction.

    Supplemental material

    Assessment of risk of bias in included trials

    For each key person, risk of bias was evaluated using the following criteria:

    • Low risk of bias: if it was mentioned that the key person was blinded, and this was sufficiently described.

    • Uncertain risk of bias: if it was not mentioned if the key person in the trial was blinded or the blinding procedures were insufficiently described.

    • High risk of bias: if no blinding or incomplete blinding of the key person was described.

    Statistical analysis

    Descriptive data were summarised as proportions with 95% CIs using Stata V.1621 command ‘prtest’ using the Wilson method.22 The CI refers to the level of uncertainty. The 95% CI represents that in a hypothetical indefinite data collection, the interval estimate will contain the true value in 95% of the samples.

    Patient and public involvement

    Patients and members of the public were not involved in this research because it was designed to answer a methodological challenge that was not directly dependent on patient priorities or experiences. The methodological expertise required to plan the study, analyse the results and write the manuscript was dependent on specialist knowledge. Hence, we did not try to identify patients or members of the public with these specialties for cooperation on this manuscript.

    Results

    A total of 63 randomised clinical trials of psychological interventions were identified.23–85 Characteristics of the included trials may be found in online supplemental file 2. An overview of results can be found in tables 1 and 2.

    View this table:
    Table 1

    Risk-of-bias assessment of blinding status in 63 randomised clinical trials of psychological interventions

    View this table:
    Table 2

    Handling of results in non-blinded trials

    Blinding of all possible parties

    None (0%; 95% CI 0% to 5.75%) of the identified trials reported blinding of all possible parties, that is, the ‘true’ proportion of trials with blinding of all possible parties is probably between 0% and 5.75%.

    Blinding of participants

    Two (3.2%; 95% CI 0.87% to 10.86%) of the 63 trials reported adequate blinding of participants24 67 (table 1). One trial compared the effects of internet-delivered cognitive–behavioural therapy with internet patient education.24 There were no treatment providers involved. The other trial compared the effects of gaze-contingent versus gaze-non-contingent music reward therapy.67 In both trials, participants were blinded to their allocated intervention, as they theoretically could not distinguish the experimental condition from the control. The remaining trials either did not adequately report if blinding of participants was performed and the bias risk was then rated as ‘unclear’, or they reported that participants were unblinded and the bias risk for was then rated as ‘high’.

    Blinding of treatment providers

    None (0%; 95% CI 0% to 5.83%) of the 62 applicable trials reported blinding of treatment providers (table 1). One trial was not applicable for assessment in this domain, as both interventions were purely internet delivered, and thus did not have any treatment providers.24 The remaining trials either did not adequately report if blinding of treatment providers was performed, and the bias risk was then rated as unclear, or they reported that treatment providers were unblinded, and the bias risk for was then rated as high.

    Blinding of outcome assessors

    37 (58.7%; 95% CI 46.42% to 70.04%) of the 63 trials reported adequate blinding of outcome assessors on the primary outcome (table 1). The remaining trials either did not adequately report if blinding of outcome assessors was performed, and the bias risk was then rated as unclear, or they reported that outcome assessors were unblinded, and the bias risk for was then rated as high. When the primary outcome was purely participant reported, the risk of bias was equivalent to the risk of bias for the participant domain, for example, the outcome assessor domain was assessed as high, if the participant was unblinded while filling in the questionnaire. If trialists assessed more than one primary outcome, and these were assessed at different risks of bias, we chose the primary outcome with the highest risk of bias.

    Blinding of data managers

    Two (3.2%; 95% CI 0.87% to 10.86%) of the 63 trials reported adequate blinding of data managers48 62 (table 1). One trial reported that: “[d]ata coding, data entry, and data cleaning were done by individuals masked to treatment allocation”.48 The other trial reported that “…data managers were masked to study allocation”.62 Two trials reported that the data manager was unblinded.68 81 The bias risk for this domain was then rated as high. The remaining trials did not mention if blinding of data managers was performed and the bias risk was then rated as unclear.

    Blinding of the data safety and monitoring committee

    None (0%; 95% CI 0% to 16.11%) of the 20 eligible trials reported blinding of the data safety and monitoring committee (table 1). Forty-three trials were not eligible for assessment in this domain, as they did not include a data safety and monitoring committee. In the 20 eligible trials, the blinding status of the data safety and monitoring committee was not described and the bias risk was then rated as unclear.

    Blinding of statisticians

    Three (4.8%; 95% CI 1.63% to 13.09%) of the 63 trials reported adequate blinding of statisticians during data analysis25 76 81 (table 1). One trial reported that “[i]ndependent statisticians analysing the results were masked to group for the initial analyses”.76 The second trial reported that “The research assistant for data analysis was blinded in respect of the group allocation results”.81 The third trial reported that “the trial statistician was blinded to the allocated treatments during the analysis of the primary outcome”.25 Three trials reported that the statistician was unblinded while analysing data.34 39 47 The bias risk for this domain was then rated as high. The remaining trials did not mention if blinding of statisticians was performed during data analysis, and the bias risk was then rated as unclear.

    Blinding of conclusion makers

    None (0%; 95% CI 0% to 5.75%) of the 63 trials reported adequate blinding of conclusion makers (table 1). One trial reported that interpretation of results was performed on unmasked data.40 The bias risk for this domain was then rated as high. The remaining trials did not mention if blinding of conclusion makers was performed, and the bias risk was then rated as unclear.

    Rejection of null hypothesis

    Forty-five (71.4%; 95% CI 59.30% to 81.10%) of the 63 trials rejected the null hypothesis on the primary outcome(s), meaning that statistical superiority of the experimental intervention was found in a superiority trial, equivalence was found in equivalence trials and non-inferiority was found in non-inferiority trials (table 2). We assessed a rejection of the null hypothesis if the trialists rejected the null hypothesis on at least one of the primary outcomes. In 67.5% of the trials that blinded the outcome assessor, the null hypothesis was rejected on the primary outcome. In 76.9% of the trials that did not blind the outcome assessor, or where the blinding status of the outcome assessor was unclear, the null hypothesis was rejected on the primary outcome.

    Discussion of bias risk due to lack of blinding in the published trial report

    Thirteen (20.6%; 95% CI 12.48% to 32.17%) of the 63 trials discussed the potential bias risk from lack of blinding in the published trial report24 25 29 31 39 49 56 57 60 62 67 70 84 (table 2). Seven trials discussed the bias risk from lack of participant blinding.24 25 31 49 56 60 67 Two trials discussed the bias risk from lack of therapist blinding.29 84 Two trials discussed the bias risk from lack of outcome assessor blinding.57 70 One trial discussed the bias risk from lack of researcher blinding.39 One trial discussed the bias risk from lack of ‘full blinding’.62

    Discussion

    Blinding of all possible key persons is rarely reported as adequate in randomised clinical trials of psychological interventions. Some key persons are more frequently blinded than others. In the present study of trials published in high impact factor journals, 58.7% of the included trials reported adequate blinding of outcome assessors. Participants, treatment providers, data managers, the data safety and monitoring committee, statisticians and conclusion makers were typically either not blinded (high risk of bias) or it was unclear whether blinding was performed (unclear risk of bias). However, it may be inappropriate to assume inadequate blinding merely based on inadequate reporting. Several studies indicate that trialists often do not report blinding procedures even when such procedures have been adequately implemented.86–88 Nevertheless, accurate reporting of blinding efforts and discussions of lack of blinding may reduce the risk of biased conclusions. In the present study, only 20.6% of the trials discussed the potential bias risk from unsuccessful or lack of blinding in the published trial report. This makes it difficult for readers to judge the quality of the research. 71.4% of the trials rejected their null hypothesis on their primary outcome. Considering our findings, there is a risk that previous randomised clinical trials of psychological interventions may have overestimated the beneficial effects and underestimated the harmful effects of the experimental interventions being studied due to bias risks associated with lack of blinding.2–9 There may be reasons to believe that unblinded trial key persons may, consciously or unconsciously, fail to acknowledge harmful effects of the interventions. A non-blinded trial person who has a certain interest in a result, for example a psychotherapist with many years of experience and expertise in a given psychological intervention, might give less attention to harmful effects because of his or hers underlying beliefs. Likewise, an unblinded participant who is told that the psychological intervention is effective and without any harms might either not register or fail to report harmful effects.

    The present study has limitations. First, we did not publish a protocol and a statistical analysis plan prior to this retrospective study of published trial reports. Second, we may have missed important trials with sufficient blinding due to our selection of journals. However, journals with high journal impact factor are known to have lower risks of bias compared to journals with lower journal impact factor.89 90 Hence, there is a possibility that the included trials from the selected journals may have underestimated the true bias risk arising from lack of blinding, and that blinding is in fact even less frequently performed or reported if trials from journals with lower journal impact factor had also been included. Third, we only looked at psychological interventions although other intervention areas (eg, pharmacological interventions) may have comparable problems with blinding.

    It may be argued that it is difficult or impossible to blind all relevant key persons in a randomised clinical trial. In pharmacological trials, participants may experience adverse events (or lack of adverse events) which may break their blinding.91–94 In non-pharmacological trials, for example, surgical trials, it is challenging to perform sham interventions for blinding purposes.95–97 Nevertheless, blinding of all possible key persons have been attempted in previous trials.98 99 It is possible that the blinding of some of the key persons in these trials was not successful, but adequate blinding was attempted. It has been debated whether it may be possible to test the success of blinding by having, for example, the outcome assessors guess the participant’s allocated intervention on completion of the assessment interview.94 100 101 However, it is generally recommended not to assess whether blinding procedures were successful, because these tests lack validity, as they cannot distinguish blindness from hunches about harms, adverse effects or efficacy.92 100 102

    Whether or not blinding of participants and treatment providers is a bias risk that needs to be controlled for in randomised clinical trials of psychological interventions must be discussed. In trials investigating psychotherapy (as opposed to drug trials), it is argued that it is redundant to control for self-confirming response expectancies, given that expectancy change is a legitimate, potentially effective psychological variable.103–105 For example, changing expectations is a core feature of cognitive–behavioural therapy. Thus, one could argue that it would be counterproductive to attempt adequate blinding of participants and treatment providers, as it would diminish the effective ingredients of the interventions being studied. Furthermore, there would be substantial practical problems in such trial, as it would require training of motivated potential therapists with no or limited prior experience in psychotherapy, who would then deliver the intervention without knowing the exact type of psychotherapy tradition while maintaining blinding. Likewise, it would require recruitment of participants with no prior experience and knowledge of psychotherapy traditions.

    Despite this apparent obstacle, one could argue that it is appropriate to expect bias due to non-blinded participants. Research in non-psychological interventions has demonstrated that non-blinded participants may experience and report symptoms differently from blinded ones, because of response bias (when participants report symptoms according to what they think will please the investigators) and because of positive response expectancy from receiving a treatment considered to be superior.1 103 104 In a randomised clinical trial, this could result in participants (consciously or unconsciously) giving exaggerated reports of symptom relief merely because the treatment providers or outcome assessors are perceived as caring and interested in their well-being. It may also produce accurate reports of greater symptom relief, because of self-confirming response expectancies.103 A systematic review found that non-blinded participants generated more optimistic self-reported estimates of intervention effects compared with blinded ones.7 There is no reason to believe, that these processes would not also translate to randomised clinical trials of psychological interventions. Non-blinded participants in psychological trials may, for example, search the internet for information about the psychological intervention they receive, which may influence their outcomes, because they develop an allegiance to the specific psychological tradition. This can become particularly challenging in the case of participant-reported outcomes, for example, symptoms or quality of life, where the participant is also the outcome assessor.18

    In randomised clinical trials of psychological interventions, it may also be appropriate to expect bias due to non-blinded treatment providers.1 Therapists are often highly trained and supervised in delivering a specific psychological intervention, and they will then probably expect this intervention to be superior to treatment as usual given their personal investment. This expectation may influence the participants’ outcomes, if the therapists disclose their allegiance to the participants. It may even influence the participants’ outcomes even if the therapists do not disclose their allegiance, given that expectation of improvement can alter therapists’ behaviours in ways that alter the effectiveness of the treatment.

    It is possible to design a psychotherapy trial attempting to blind participants and treatment providers. The first step would be to randomise motivated potential therapists with no or limited knowledge of psychotherapy to receive either blinded or non-blinded training and supervision and subsequently deliver blinded or non-blinded psychotherapy. If the number of therapists is high enough, it will be possible to assess if attempts of blinded psychotherapy would yield different intervention effects compared with psychotherapy delivered by non-blinded therapists, and hence whether attempting blinding of therapists and participants compromises the active ingredients of psychotherapy. If no evidence of a difference is found, the second step would be to design a randomised clinical trial in which two psychotherapy traditions were compared with attempts of blinding of all possible key persons. Of course, such trial would have limitations, for example, effective delivery of the psychological intervention would potentially require extensive training of the therapists, which would be difficult to implement within a trial period while keeping the blinding intact. Thus, there would be a risk that the lower amount of training would reduce generalisability of the findings. Recruiting participants with no or limited knowledge and experience about psychological interventions could also reduce generalisability. Furthermore, the proposed methods to blind the participants could theoretically work only in trials with an ‘active’ control group. If the control group is standard care, there would be a risk that the participants would recognise the provided care from previous experience. Nevertheless, it may be argued that the benefits in terms of complete blinding may not exceed the limitations arising from low generalisability. Until evidence has shown that blinding does not significantly affect the results, we believe that blinding of all possible parties should ideally be attempted in randomised clinical trials of psychological interventions.

    Conclusions

    Blinding of key persons involved is rarely documented in randomised clinical trials of psychological interventions, and only few trialists consider the bias risk this may have caused in trials published in high impact factor journals. Therefore, there is a risk that previous randomised clinical trials of psychological interventions generally may have overestimated the beneficial effects and underestimated the harmful effects of the experimental interventions being studied. There is a need of randomised clinical trials assessing the effects of psychological interventions with attempts of blinding of all possible key persons. If blinding is not possible to implement, or is not adequately reported, readers should consider the possible implications when interpreting the trial results. Future randomised clinical trials of psychological interventions should improve implementation and reporting of blinding status, particularly of trial persons who are easily blinded, that is, data managers, the data safety and monitoring committee, statisticians and conclusion makers.

    Data availability statement

    All data relevant to the study are included in the article or uploaded as supplementary information.

    References

    1. ↵
      2. Hróbjartsson A ,
      3. Boutron I
      . Blinding in randomized clinical trials: imposed impartiality. Clin Pharmacol Ther 2011;90:732–6.doi:10.1038/clpt.2011.207 pmid:http://www.ncbi.nlm.nih.gov/pubmed/21993424
      OpenUrlCrossRefPubMed
    2. ↵
      2. Gluud LL
      . Bias in clinical intervention research. Am J Epidemiol 2006;163:493–501.doi:10.1093/aje/kwj069 pmid:http://www.ncbi.nlm.nih.gov/pubmed/16443796
      OpenUrlCrossRefPubMedWeb of Science
    3. ↵
      2. Kjaergard LL ,
      3. Villumsen J ,
      4. Gluud C
      . Reported methodologic quality and discrepancies between large and small randomized trials in meta-analyses. Ann Intern Med 2001;135:982–9.doi:10.7326/0003-4819-135-11-200112040-00010 pmid:http://www.ncbi.nlm.nih.gov/pubmed/11730399
      OpenUrlCrossRefPubMedWeb of Science
    4. ↵
      2. Jüni P ,
      3. Altman DG ,
      4. Egger M
      . Assessing the quality of controlled clinical trials. BMJ 2001;323:42–6.
      OpenUrlFREE Full Text
    5. ↵
      2. Moher D ,
      3. Pham Ba' ,
      4. Jones A , et al
      . Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet 1998;352:609–13.doi:10.1016/S0140-6736(98)01085-X
      OpenUrlCrossRefPubMedWeb of Science
    6. ↵
      2. Schulz KF ,
      3. Chalmers I ,
      4. Hayes RJ , et al
      . Empirical evidence of bias. dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995;273:408–12.doi:10.1001/jama.273.5.408 pmid:http://www.ncbi.nlm.nih.gov/pubmed/7823387
      OpenUrlCrossRefPubMedWeb of Science
    7. ↵
      2. Hróbjartsson A ,
      3. Emanuelsson F ,
      4. Skou Thomsen AS , et al
      . Bias due to lack of patient blinding in clinical trials. A systematic review of trials randomizing patients to blind and nonblind sub-studies. Int J Epidemiol 2014;43:1272–83.doi:10.1093/ije/dyu115 pmid:http://www.ncbi.nlm.nih.gov/pubmed/24881045
      OpenUrlCrossRefPubMed
    8. ↵
      2. Hróbjartsson A ,
      3. Thomsen ASS ,
      4. Emanuelsson F , et al
      . Observer bias in randomized clinical trials with measurement scale outcomes: a systematic review of trials with both blinded and nonblinded assessors. CMAJ 2013;185:E201–11.doi:10.1503/cmaj.120744 pmid:http://www.ncbi.nlm.nih.gov/pubmed/23359047
      OpenUrlAbstract/FREE Full Text
    9. ↵
      2. Hrobjartsson A ,
      3. Thomsen ASS ,
      4. Emanuelsson F , et al
      . Observer bias in randomised clinical trials with binary outcomes: systematic review of trials with both blinded and non-blinded outcome assessors. BMJ 2012;344:e1119. doi:10.1136/bmj.e1119
    10. ↵
      2. Savović J ,
      3. Turner RM ,
      4. Mawdsley D , et al
      . Association between risk-of-bias assessments and results of randomized trials in Cochrane reviews: the ROBES Meta-Epidemiologic study. Am J Epidemiol 2018;187:1113–22.doi:10.1093/aje/kwx344 pmid:http://www.ncbi.nlm.nih.gov/pubmed/29126260
      OpenUrlCrossRefPubMed
    11. ↵
      2. Balk EM ,
      3. Bonis PAL ,
      4. Moskowitz H , et al
      . Correlation of quality measures with estimates of treatment effect in meta-analyses of randomized controlled trials. JAMA 2002;287:2973–82.doi:10.1001/jama.287.22.2973 pmid:http://www.ncbi.nlm.nih.gov/pubmed/12052127
      OpenUrlCrossRefPubMedWeb of Science
    12. ↵
      2. Wood L ,
      3. Egger M ,
      4. Gluud LL , et al
      . Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ 2008;336:601–5.doi:10.1136/bmj.39465.451748.AD pmid:http://www.ncbi.nlm.nih.gov/pubmed/18316340
      OpenUrlAbstract/FREE Full Text
    13. ↵
      2. Moustgaard H ,
      3. Clayton GL ,
      4. Jones HE , et al
      . Impact of blinding on estimated treatment effects in randomised clinical trials: meta-epidemiological study. BMJ 2020;51:l6802.doi:10.1136/bmj.l6802
      OpenUrl
    14. ↵
      2. Drucker AM ,
      3. Chan A-W
      . Blindsided: challenging the dogma of masking in clinical trials. BMJ 2020;368:m229.doi:10.1136/bmj.m229 pmid:http://www.ncbi.nlm.nih.gov/pubmed/32020896
      OpenUrlFREE Full Text
    15. ↵
      2. Schulz KF ,
      3. Altman DG ,
      4. Moher D , et al
      . CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials. Ann Intern Med 2010;152:726–32.doi:10.7326/0003-4819-152-11-201006010-00232 pmid:http://www.ncbi.nlm.nih.gov/pubmed/20335313
      OpenUrlCrossRefPubMedWeb of Science
    16. ↵
      2. Karanicolas PJ ,
      3. Farrokhyar F ,
      4. Bhandari M
      . Blinding: who, what, when, why, how? Can J Surg 2010;53:345–8.
      OpenUrl
    17. ↵
      2. Schulz KF ,
      3. Grimes DA
      . Blinding in randomised trials: hiding who got what. Lancet 2002;359:696–700.doi:10.1016/S0140-6736(02)07816-9 pmid:http://www.ncbi.nlm.nih.gov/pubmed/11879884
      OpenUrlCrossRefPubMedWeb of Science
    18. ↵
      2. Higgins J ,
      3. Thomas J ,
      4. Chandler J , et al
      . Cochrane Handbook for systematic reviews of interventions version 6.0 (updated July 2019), 2019. Available: www.training.cochrane.org/handbook
    19. ↵
      2. Schulz KF ,
      3. Chalmers I ,
      4. Altman DG
      . The landscape and lexicon of blinding in randomized trials. Ann Intern Med 2002;136:254–9.doi:10.7326/0003-4819-136-3-200202050-00022 pmid:http://www.ncbi.nlm.nih.gov/pubmed/11827510
      OpenUrlCrossRefPubMedWeb of Science
    20. ↵
      2. Montori VM ,
      3. Bhandari M ,
      4. Devereaux PJ , et al
      . In the dark: the reporting of blinding status in randomized controlled trials. J Clin Epidemiol 2002;55:787–90.doi:10.1016/S0895-4356(02)00446-8 pmid:http://www.ncbi.nlm.nih.gov/pubmed/12384193
      OpenUrlCrossRefPubMedWeb of Science
    21. ↵
      1. StataCorp
      . Stata statistical software: release 16. College Station, TX: StataCorp LLC, 2019.
    22. ↵
      2. Wilson EB
      . Probable inference, the law of succession, and statistical inference. J Am Stat Assoc 1927;22:209–12.doi:10.1080/01621459.1927.10502953
      OpenUrlCrossRefWeb of Science
    23. ↵
      2. Resick PA ,
      3. Wachen JS ,
      4. Dondanville KA , et al
      . Effect of group vs individual cognitive processing therapy in active-duty military seeking treatment for posttraumatic stress disorder: a randomized clinical trial. JAMA Psychiatry 2017;74:28–36.doi:10.1001/jamapsychiatry.2016.2729 pmid:27893032
      OpenUrlPubMed
    24. ↵
      2. Ritterband LM ,
      3. Thorndike FP ,
      4. Ingersoll KS , et al
      . Effect of a web-based cognitive behavior therapy for insomnia intervention with 1-year follow-up: a randomized clinical trial. JAMA Psychiatry 2017;74:68–75.doi:10.1001/jamapsychiatry.2016.3249 pmid:http://www.ncbi.nlm.nih.gov/pubmed/27902836
      OpenUrlPubMed
    25. ↵
      2. Schramm E ,
      3. Kriston L ,
      4. Zobel I , et al
      . Effect of disorder-specific vs nonspecific psychotherapy for chronic depression: a randomized clinical trial. JAMA Psychiatry 2017;74:233–42.doi:10.1001/jamapsychiatry.2016.3880 pmid:http://www.ncbi.nlm.nih.gov/pubmed/28146251
      OpenUrlPubMed
    26. ↵
      2. Weersing VR ,
      3. Brent DA ,
      4. Rozenman MS , et al
      . Brief behavioral therapy for pediatric anxiety and depression in primary care: a randomized clinical trial. JAMA Psychiatry 2017;74:571–8.doi:10.1001/jamapsychiatry.2017.0429 pmid:http://www.ncbi.nlm.nih.gov/pubmed/28423145
      OpenUrlPubMed
    27. ↵
      2. Silverstein M ,
      3. Diaz-Linhart Y ,
      4. Cabral H , et al
      . Efficacy of a maternal depression prevention strategy in head start: a randomized clinical trial. JAMA Psychiatry 2017;74:781–9.doi:10.1001/jamapsychiatry.2017.1001 pmid:http://www.ncbi.nlm.nih.gov/pubmed/28614554
      OpenUrlPubMed
    28. ↵
      2. Barlow DH ,
      3. Farchione TJ ,
      4. Bullis JR , et al
      . The unified protocol for transdiagnostic treatment of emotional disorders compared with diagnosis-specific protocols for anxiety disorders: a randomized clinical trial. JAMA Psychiatry 2017;74:875–84.doi:10.1001/jamapsychiatry.2017.2164 pmid:http://www.ncbi.nlm.nih.gov/pubmed/28768327
      OpenUrlPubMed
    29. ↵
      2. de Zwaan M ,
      3. Herpertz S ,
      4. Zipfel S , et al
      . Effect of internet-based guided self-help vs individual face-to-face treatment on full or subsyndromal binge eating disorder in overweight or obese patients: the INTERBED randomized clinical trial. JAMA Psychiatry 2017;74:987–95.doi:10.1001/jamapsychiatry.2017.2150 pmid:28768334
      OpenUrlPubMed
    30. ↵
      2. Sirey JA ,
      3. Banerjee S ,
      4. Marino P , et al
      . Adherence to depression treatment in primary care: a randomized clinical trial. JAMA Psychiatry 2017;74:1129–35.doi:10.1001/jamapsychiatry.2017.3047 pmid:http://www.ncbi.nlm.nih.gov/pubmed/28973066
      OpenUrlPubMed
    31. ↵
      2. Rollman BL ,
      3. Herbeck Belnap B ,
      4. Abebe KZ , et al
      . Effectiveness of online collaborative care for treating mood and anxiety disorders in primary care: a randomized clinical trial. JAMA Psychiatry 2018;75:56–64.doi:10.1001/jamapsychiatry.2017.3379 pmid:http://www.ncbi.nlm.nih.gov/pubmed/29117275
      OpenUrlPubMed
    32. ↵
      2. Sloan DM ,
      3. Marx BP ,
      4. Lee DJ , et al
      . A brief exposure-based treatment vs cognitive processing therapy for posttraumatic stress disorder: a randomized noninferiority clinical trial. JAMA Psychiatry 2018;75:233–9.doi:10.1001/jamapsychiatry.2017.4249 pmid:http://www.ncbi.nlm.nih.gov/pubmed/29344631
      OpenUrlPubMed
    33. ↵
      2. McCauley E ,
      3. Berk MS ,
      4. Asarnow JR , et al
      . Efficacy of dialectical behavior therapy for adolescents at high risk for suicide: a randomized clinical trial. JAMA Psychiatry 2018;75:777–85.doi:10.1001/jamapsychiatry.2018.1109 pmid:http://www.ncbi.nlm.nih.gov/pubmed/29926087
      OpenUrlPubMed
    34. ↵
      2. Shawyer F ,
      3. Farhall J ,
      4. Thomas N , et al
      . Acceptance and commitment therapy for psychosis: randomised controlled trial. Br J Psychiatry 2017;210:140–8.doi:10.1192/bjp.bp.116.182865 pmid:http://www.ncbi.nlm.nih.gov/pubmed/27979820
      OpenUrlAbstract/FREE Full Text
    35. ↵
      2. Gilbody S ,
      3. Brabyn S ,
      4. Lovell K , et al
      . Telephone-supported computerised cognitive-behavioural therapy: REEACT-2 large-scale pragmatic randomised controlled trial. Br J Psychiatry 2017;210:362–7.doi:10.1192/bjp.bp.116.192435 pmid:http://www.ncbi.nlm.nih.gov/pubmed/28254959
      OpenUrlAbstract/FREE Full Text
    36. ↵
      2. Pearce S ,
      3. Scott L ,
      4. Attwood G , et al
      . Democratic therapeutic community treatment for personality disorder: randomised controlled trial. Br J Psychiatry 2017;210:149–56.doi:10.1192/bjp.bp.116.184366 pmid:http://www.ncbi.nlm.nih.gov/pubmed/27908900
      OpenUrlAbstract/FREE Full Text
    37. ↵
      2. Bouchard S ,
      3. Dumoulin S ,
      4. Robillard G , et al
      . Virtual reality compared with in vivo exposure in the treatment of social anxiety disorder: a three-arm randomised controlled trial. Br J Psychiatry 2017;210:276–83.doi:10.1192/bjp.bp.116.184234 pmid:http://www.ncbi.nlm.nih.gov/pubmed/27979818
      OpenUrlAbstract/FREE Full Text
    38. ↵
      2. Cooney P ,
      3. Jackman C ,
      4. Coyle D , et al
      . Computerised cognitive-behavioural therapy for adults with intellectual disability: randomised controlled trial. Br J Psychiatry 2017;211:95–102.doi:10.1192/bjp.bp.117.198630 pmid:http://www.ncbi.nlm.nih.gov/pubmed/28596245
      OpenUrlAbstract/FREE Full Text
    39. ↵
      2. Williams C ,
      3. McClay C-A ,
      4. Matthews L , et al
      . Community-based group guided self-help intervention for low mood and stress: randomised controlled trial. Br J Psychiatry 2018;212:88–95.doi:10.1192/bjp.2017.18 pmid:http://www.ncbi.nlm.nih.gov/pubmed/29436324
      OpenUrlPubMed
    40. ↵
      2. Janse A ,
      3. Worm-Smeitink M ,
      4. Bleijenberg G , et al
      . Efficacy of web-based cognitive-behavioural therapy for chronic fatigue syndrome: randomised controlled trial. Br J Psychiatry 2018;212:112–8.doi:10.1192/bjp.2017.22 pmid:http://www.ncbi.nlm.nih.gov/pubmed/29436329
      OpenUrlPubMed
    41. ↵
      2. Stigsdotter UK ,
      3. Corazon SS ,
      4. Sidenius U , et al
      . Efficacy of nature-based therapy for individuals with stress-related illnesses: randomised controlled trial. Br J Psychiatry 2018;213:404–11.doi:10.1192/bjp.2018.2 pmid:http://www.ncbi.nlm.nih.gov/pubmed/29793558
      OpenUrlPubMed
    42. ↵
      2. Rossouw J ,
      3. Yadin E ,
      4. Alexander D , et al
      . Prolonged exposure therapy and supportive counselling for post-traumatic stress disorder in adolescents: task-shifting randomised controlled trial. Br J Psychiatry 2018;213:587–94.doi:10.1192/bjp.2018.130 pmid:http://www.ncbi.nlm.nih.gov/pubmed/29991358
      OpenUrlPubMed
    43. ↵
      2. Camacho EM ,
      3. Davies LM ,
      4. Hann M , et al
      . Long-term clinical and cost-effectiveness of collaborative care (versus usual care) for people with mental-physical multimorbidity: cluster-randomised trial. Br J Psychiatry 2018;213:456–63.doi:10.1192/bjp.2018.70 pmid:http://www.ncbi.nlm.nih.gov/pubmed/29761751
      OpenUrlPubMed
    44. ↵
      2. Bockting CLH ,
      3. Klein NS ,
      4. Elgersma HJ , et al
      . Effectiveness of preventive cognitive therapy while tapering antidepressants versus maintenance antidepressant treatment versus their combination in prevention of depressive relapse or recurrence (DRD study): a three-group, multicentre, randomised controlled trial. Lancet Psychiatry 2018;5:401–10.doi:10.1016/S2215-0366(18)30100-7 pmid:http://www.ncbi.nlm.nih.gov/pubmed/29625762
      OpenUrlPubMed
    45. ↵
      2. Cottrell DJ ,
      3. Wright-Hughes A ,
      4. Collinson M , et al
      . Effectiveness of systemic family therapy versus treatment as usual for young people after self-harm: a pragmatic, phase 3, multicentre, randomised controlled trial. Lancet Psychiatry 2018;5:203–16.doi:10.1016/S2215-0366(18)30058-0 pmid:http://www.ncbi.nlm.nih.gov/pubmed/29449180
      OpenUrlPubMed
    46. ↵
      2. Craig TK ,
      3. Rus-Calafell M ,
      4. Ward T , et al
      . AVATAR therapy for auditory verbal hallucinations in people with psychosis: a single-blind, randomised controlled trial. Lancet Psychiatry 2018;5:31–40.doi:10.1016/S2215-0366(17)30427-3 pmid:http://www.ncbi.nlm.nih.gov/pubmed/29175276
      OpenUrlPubMed
    47. ↵
      2. Creswell C ,
      3. Violato M ,
      4. Fairbanks H , et al
      . Clinical outcomes and cost-effectiveness of brief guided parent-delivered cognitive behavioural therapy and solution-focused brief therapy for treatment of childhood anxiety disorders: a randomised controlled trial. Lancet Psychiatry 2017;4:529–39.doi:10.1016/S2215-0366(17)30149-9 pmid:http://www.ncbi.nlm.nih.gov/pubmed/28527657
      OpenUrlPubMed
    48. ↵
      2. Fonagy P ,
      3. Butler S ,
      4. Cottrell D , et al
      . Multisystemic therapy versus management as usual in the treatment of adolescent antisocial behaviour (START): a pragmatic, randomised controlled, superiority trial. Lancet Psychiatry 2018;5:119–33.doi:10.1016/S2215-0366(18)30001-4 pmid:http://www.ncbi.nlm.nih.gov/pubmed/29307527
      OpenUrlPubMed
    49. ↵
      2. Fowler D ,
      3. Hodgekins J ,
      4. French P , et al
      . Social recovery therapy in combination with early intervention services for enhancement of social recovery in patients with first-episode psychosis (SUPEREDEN3): a single-blind, randomised controlled trial. Lancet Psychiatry 2018;5:41–50.doi:10.1016/S2215-0366(17)30476-5 pmid:http://www.ncbi.nlm.nih.gov/pubmed/29242000
      OpenUrlPubMed
    50. ↵
      2. Freeman D ,
      3. Sheaves B ,
      4. Goodwin GM , et al
      . The effects of improving sleep on mental health (OASIS): a randomised controlled trial with mediation analysis. Lancet Psychiatry 2017;4:749–58.doi:10.1016/S2215-0366(17)30328-0 pmid:http://www.ncbi.nlm.nih.gov/pubmed/28888927
      OpenUrlPubMed
    51. ↵
      2. Goodyer IM ,
      3. Reynolds S ,
      4. Barrett B , et al
      . Cognitive behavioural therapy and short-term psychoanalytical psychotherapy versus a brief psychosocial intervention in adolescents with unipolar major depressive disorder (IMPACT): a multicentre, pragmatic, observer-blind, randomised controlled superiority trial. Lancet Psychiatry 2017;4:109–19.doi:10.1016/S2215-0366(16)30378-9 pmid:http://www.ncbi.nlm.nih.gov/pubmed/27914903
      OpenUrlPubMed
    52. ↵
      2. Jahoda A ,
      3. Hastings R ,
      4. Hatton C , et al
      . Comparison of behavioural activation with guided self-help for treatment of depression in adults with intellectual disabilities: a randomised controlled trial. Lancet Psychiatry 2017;4:909–19.doi:10.1016/S2215-0366(17)30426-1 pmid:http://www.ncbi.nlm.nih.gov/pubmed/29153873
      OpenUrlPubMed
    53. ↵
      2. O'Connor RC ,
      3. Ferguson E ,
      4. Scott F , et al
      . A brief psychological intervention to reduce repetition of self-harm in patients admitted to hospital following a suicide attempt: a randomised controlled trial. Lancet Psychiatry 2017;4:451–60.doi:10.1016/S2215-0366(17)30129-3 pmid:http://www.ncbi.nlm.nih.gov/pubmed/28434871
      OpenUrlPubMed
    54. ↵
      2. Pot-Kolder RMCA ,
      3. Geraets CNW ,
      4. Veling W , et al
      . Virtual-reality-based cognitive behavioural therapy versus waiting list control for paranoid ideation and social avoidance in patients with psychotic disorders: a single-blind randomised controlled trial. Lancet Psychiatry 2018;5:217–26.doi:10.1016/S2215-0366(18)30053-1 pmid:http://www.ncbi.nlm.nih.gov/pubmed/29429948
      OpenUrlPubMed
    55. ↵
      2. Schönenberg M ,
      3. Wiedemann E ,
      4. Schneidt A , et al
      . Neurofeedback, sham neurofeedback, and cognitive-behavioural group therapy in adults with attention-deficit hyperactivity disorder: a triple-blind, randomised, controlled trial. Lancet Psychiatry 2017;4:673–84.doi:10.1016/S2215-0366(17)30291-2 pmid:http://www.ncbi.nlm.nih.gov/pubmed/28803030
      OpenUrlPubMed
    56. ↵
      2. Morrison AP ,
      3. Law H ,
      4. Carter L , et al
      . Antipsychotic drugs versus cognitive behavioural therapy versus a combination of both in people with psychosis: a randomised controlled pilot and feasibility study. Lancet Psychiatry 2018;5:411–23.doi:10.1016/S2215-0366(18)30096-8 pmid:http://www.ncbi.nlm.nih.gov/pubmed/29605187
      OpenUrlPubMed
    57. ↵
      2. Arjadi R ,
      3. Nauta MH ,
      4. Scholte WF , et al
      . Internet-based behavioural activation with lay counsellor support versus online minimal psychoeducation without support for treatment of depression: a randomised controlled trial in Indonesia. Lancet Psychiatry 2018;5:707–16.doi:10.1016/S2215-0366(18)30223-2 pmid:http://www.ncbi.nlm.nih.gov/pubmed/30006262
      OpenUrlPubMed
    58. ↵
      2. Delgadillo J ,
      3. de Jong K ,
      4. Lucock M , et al
      . Feedback-informed treatment versus usual psychological treatment for depression and anxiety: a multisite, open-label, cluster randomised controlled trial. Lancet Psychiatry 2018;5:564–72.doi:10.1016/S2215-0366(18)30162-7 pmid:http://www.ncbi.nlm.nih.gov/pubmed/29937396
      OpenUrlPubMed
    59. ↵
      2. Freeman D ,
      3. Haselton P ,
      4. Freeman J , et al
      . Automated psychological therapy using immersive virtual reality for treatment of fear of heights: a single-blind, parallel-group, randomised controlled trial. Lancet Psychiatry 2018;5:625–32.doi:10.1016/S2215-0366(18)30226-8 pmid:http://www.ncbi.nlm.nih.gov/pubmed/30007519
      OpenUrlPubMed
    60. ↵
      2. Morrison AP ,
      3. Pyle M ,
      4. Gumley A , et al
      . Cognitive behavioural therapy in clozapine-resistant schizophrenia (FOCUS): an assessor-blinded, randomised controlled trial. Lancet Psychiatry 2018;5:633–43.doi:10.1016/S2215-0366(18)30184-6 pmid:http://www.ncbi.nlm.nih.gov/pubmed/30001930
      OpenUrlPubMed
    61. ↵
      2. Nidich S ,
      3. Mills PJ ,
      4. Rainforth M , et al
      . Non-trauma-focused meditation versus exposure therapy in veterans with post-traumatic stress disorder: a randomised controlled trial. Lancet Psychiatry 2018;5:975–86.doi:10.1016/S2215-0366(18)30384-5 pmid:http://www.ncbi.nlm.nih.gov/pubmed/30449712
      OpenUrlPubMed
    62. ↵
      2. Ougrin D ,
      3. Corrigall R ,
      4. Poole J , et al
      . Comparison of effectiveness and cost-effectiveness of an intensive community supported discharge service versus treatment as usual for adolescents with psychiatric emergencies: a randomised controlled trial. Lancet Psychiatry 2018;5:477–85.doi:10.1016/S2215-0366(18)30129-9 pmid:http://www.ncbi.nlm.nih.gov/pubmed/29731412
      OpenUrlPubMed
    63. ↵
      2. Stein A ,
      3. Netsi E ,
      4. Lawrence PJ , et al
      . Mitigating the effect of persistent postnatal depression on child outcomes through an intervention to treat depression and improve parenting: a randomised controlled trial. Lancet Psychiatry 2018;5:134–44.doi:10.1016/S2215-0366(18)30006-3 pmid:http://www.ncbi.nlm.nih.gov/pubmed/29413138
      OpenUrlPubMed
    64. ↵
      2. Bormann JE ,
      3. Thorp SR ,
      4. Smith E , et al
      . Individual treatment of posttraumatic stress disorder using mantram repetition: a randomized clinical trial. Am J Psychiatry 2018;175:979–88.doi:10.1176/appi.ajp.2018.17060611 pmid:http://www.ncbi.nlm.nih.gov/pubmed/29921143
      OpenUrlPubMed
    65. ↵
      2. Fallon BA ,
      3. Ahern DK ,
      4. Pavlicova M , et al
      . A randomized controlled trial of medication and cognitive-behavioral therapy for hypochondriasis. Am J Psychiatry 2017;174:756–64.doi:10.1176/appi.ajp.2017.16020189 pmid:http://www.ncbi.nlm.nih.gov/pubmed/28659038
      OpenUrlCrossRefPubMed
    66. ↵
      2. Kiluk BD ,
      3. Nich C ,
      4. Buck MB , et al
      . Randomized clinical trial of computerized and clinician-delivered CBT in comparison with standard outpatient treatment for substance use disorders: primary within-treatment and follow-up outcomes. Am J Psychiatry 2018;175:853–63.doi:10.1176/appi.ajp.2018.17090978 pmid:http://www.ncbi.nlm.nih.gov/pubmed/29792052
      OpenUrlPubMed
    67. ↵
      2. Lazarov A ,
      3. Pine DS ,
      4. Bar-Haim Y
      . Gaze-contingent music reward therapy for social anxiety disorder: a randomized controlled trial. Am J Psychiatry 2017;174:649–56.doi:10.1176/appi.ajp.2016.16080894 pmid:http://www.ncbi.nlm.nih.gov/pubmed/28103714
      OpenUrlPubMed
    68. ↵
      2. Luby JL ,
      3. Barch DM ,
      4. Whalen D , et al
      . A randomized controlled trial of parent-child psychotherapy targeting emotion development for early childhood depression. Am J Psychiatry 2018;175:1102–10.doi:10.1176/appi.ajp.2018.18030321 pmid:http://www.ncbi.nlm.nih.gov/pubmed/29921144
      OpenUrlPubMed
    69. ↵
      2. Thase ME ,
      3. Wright JH ,
      4. Eells TD , et al
      . Improving the efficiency of psychotherapy for depression: computer-assisted versus standard CBT. Am J Psychiatry 2018;175:242–50.doi:10.1176/appi.ajp.2017.17010089 pmid:http://www.ncbi.nlm.nih.gov/pubmed/28969439
      OpenUrlPubMed
    70. ↵
      2. Malla A ,
      3. Joober R ,
      4. Iyer S , et al
      . Comparing three-year extension of early intervention service to regular care following two years of early intervention service in first-episode psychosis: a randomized single blind clinical trial. World Psychiatry 2017;16:278–86.doi:10.1002/wps.20456 pmid:http://www.ncbi.nlm.nih.gov/pubmed/28941098
      OpenUrlPubMed
    71. ↵
      2. Biesheuvel-Leliefeld KEM ,
      3. Dijkstra-Kersten SMA ,
      4. van Schaik DJF , et al
      . Effectiveness of supported self-help in recurrent depression: a randomized controlled trial in primary care. Psychother Psychosom 2017;86:220–30.doi:10.1159/000472260 pmid:http://www.ncbi.nlm.nih.gov/pubmed/28647744
      OpenUrlPubMed
    72. ↵
      2. Ducasse D ,
      3. Jaussent I ,
      4. Arpon-Brand V , et al
      . Acceptance and commitment therapy for the management of suicidal patients: a randomized controlled trial. Psychother Psychosom 2018;87:211–22.doi:10.1159/000488715 pmid:http://www.ncbi.nlm.nih.gov/pubmed/29874680
      OpenUrlPubMed
    73. ↵
      2. Gieselmann A ,
      3. Böckermann M ,
      4. Sorbi M , et al
      . The effects of an internet-based imagery rehearsal intervention: a randomized controlled trial. Psychother Psychosom 2017;86:231–40.doi:10.1159/000470846 pmid:28647746
      OpenUrlPubMed
    74. ↵
      2. Hyett MP ,
      3. Bank SR ,
      4. Lipp OV , et al
      . Attenuated psychophysiological reactivity following single-session group imagery rescripting versus verbal restructuring in social anxiety disorder: results from a randomized controlled trial. Psychother Psychosom 2018;87:340–9.doi:10.1159/000493897 pmid:http://www.ncbi.nlm.nih.gov/pubmed/30380535
      OpenUrlPubMed
    75. ↵
      2. Lindsäter E ,
      3. Axelsson E ,
      4. Salomonsson S , et al
      . Internet-based cognitive behavioral therapy for chronic stress: a randomized controlled trial. Psychother Psychosom 2018;87:296–305.doi:10.1159/000490742 pmid:http://www.ncbi.nlm.nih.gov/pubmed/30041167
      OpenUrlPubMed
    76. ↵
      2. McKenna L ,
      3. Marks EM ,
      4. Hallsworth CA , et al
      . Mindfulness-based cognitive therapy as a treatment for chronic tinnitus: a randomized controlled trial. Psychother Psychosom 2017;86:351–61.doi:10.1159/000478267 pmid:http://www.ncbi.nlm.nih.gov/pubmed/29131084
      OpenUrlPubMed
    77. ↵
      2. Rief W ,
      3. Bleichhardt G ,
      4. Dannehl K , et al
      . Comparing the efficacy of CBASP with two versions of CBT for depression in a routine care center: a randomized clinical trial. Psychother Psychosom 2018;87:164–78.doi:10.1159/000487893 pmid:http://www.ncbi.nlm.nih.gov/pubmed/29649805
      OpenUrlPubMed
    78. ↵
      2. Ruglass LM ,
      3. Lopez-Castro T ,
      4. Papini S , et al
      . Concurrent treatment with prolonged exposure for co-occurring full or subthreshold posttraumatic stress disorder and substance use disorders: a randomized clinical trial. Psychother Psychosom 2017;86:150–61.doi:10.1159/000462977 pmid:http://www.ncbi.nlm.nih.gov/pubmed/28490022
      OpenUrlPubMed
    79. ↵
      2. Salzer S ,
      3. Stefini A ,
      4. Kronmüller K-T , et al
      . Cognitive-behavioral and psychodynamic therapy in adolescents with social anxiety disorder: a multicenter randomized controlled trial. Psychother Psychosom 2018;87:223–33.doi:10.1159/000488990 pmid:http://www.ncbi.nlm.nih.gov/pubmed/29895001
      OpenUrlPubMed
    80. ↵
      2. von Känel R ,
      3. Barth J ,
      4. Princip M , et al
      . Early psychological counseling for the prevention of posttraumatic stress induced by acute coronary syndrome: the MI-SPRINT randomized controlled trial. Psychother Psychosom 2018;87:75–84.doi:10.1159/000486099 pmid:http://www.ncbi.nlm.nih.gov/pubmed/29462823
      OpenUrlPubMed
    81. ↵
      2. Wong SY-S ,
      3. Zhang D-X ,
      4. Li CC-K , et al
      . Comparing the effects of mindfulness-based cognitive therapy and sleep psycho-education with exercise on chronic insomnia: a randomised controlled trial. Psychother Psychosom 2017;86:241–53.doi:10.1159/000470847 pmid:http://www.ncbi.nlm.nih.gov/pubmed/28647747
      OpenUrlPubMed
    82. ↵
      2. Zagorscak P ,
      3. Heinrich M ,
      4. Sommer D , et al
      . Benefits of individualized feedback in internet-based interventions for depression: a randomized controlled trial. Psychother Psychosom 2018;87:32–45.doi:10.1159/000481515 pmid:29306945
      OpenUrlPubMed
    83. ↵
      2. Zerwas SC ,
      3. Watson HJ ,
      4. Hofmeier SM , et al
      . CBT4BN: a randomized controlled trial of online chat and face-to-face group therapy for bulimia nervosa. Psychother Psychosom 2017;86:47–53.doi:10.1159/000449025 pmid:http://www.ncbi.nlm.nih.gov/pubmed/27883997
      OpenUrlPubMed
    84. ↵
      2. Zwerenz R ,
      3. Becker J ,
      4. Knickenberg RJ , et al
      . Online self-help as an add-on to inpatient psychotherapy: efficacy of a new blended treatment approach. Psychother Psychosom 2017;86:341–50.doi:10.1159/000481177 pmid:http://www.ncbi.nlm.nih.gov/pubmed/29131090
      OpenUrlPubMed
    85. ↵
      2. A-Tjak JGL ,
      3. Morina N ,
      4. Topper M , et al
      . A randomized controlled trial in routine clinical practice comparing acceptance and commitment therapy with cognitive behavioral therapy for the treatment of major depressive disorder. Psychother Psychosom 2018;87:154–63.doi:10.1159/000486807 pmid:http://www.ncbi.nlm.nih.gov/pubmed/29566394
      OpenUrlPubMed
    86. ↵
      2. Haahr MT ,
      3. Hróbjartsson A
      . Who is blinded in randomized clinical trials? A study of 200 trials and a survey of authors. Clin Trials 2006;3:360–5.doi:10.1177/1740774506069153 pmid:http://www.ncbi.nlm.nih.gov/pubmed/17060210
      OpenUrlCrossRefPubMedWeb of Science
    87. ↵
      2. Devereaux PJ ,
      3. Choi PT-L ,
      4. El-Dika S , et al
      . An observational study found that authors of randomized controlled trials frequently use concealment of randomization and blinding, despite the failure to report these methods. J Clin Epidemiol 2004;57:1232–6.doi:10.1016/j.jclinepi.2004.03.017 pmid:http://www.ncbi.nlm.nih.gov/pubmed/15617948
      OpenUrlCrossRefPubMedWeb of Science
    88. ↵
      2. Hróbjartsson A ,
      3. Pildal J ,
      4. Chan A-W , et al
      . Reporting on blinding in trial protocols and corresponding publications was often inadequate but rarely contradictory. J Clin Epidemiol 2009;62:967–73.doi:10.1016/j.jclinepi.2009.04.003 pmid:http://www.ncbi.nlm.nih.gov/pubmed/19635403
      OpenUrlCrossRefPubMed
    89. ↵
      2. Bala MM ,
      3. Akl EA ,
      4. Sun X , et al
      . Randomized trials published in higher vs. lower impact journals differ in design, conduct, and analysis. J Clin Epidemiol 2013;66:286–95.doi:10.1016/j.jclinepi.2012.10.005
      OpenUrlCrossRefPubMed
    90. ↵
      2. Gluud LL ,
      3. Sørensen TIA ,
      4. Gøtzsche PC , et al
      . The journal impact factor as a predictor of trial quality and outcomes: cohort study of hepatobiliary randomized clinical trials. Am J Gastroenterol 2005;100:2431–5.doi:10.1111/j.1572-0241.2005.00327.x pmid:16279896
      OpenUrlCrossRefPubMedWeb of Science
    91. ↵
      2. Fergusson D ,
      3. Glass KC ,
      4. Waring D , et al
      . Turning a blind eye: the success of blinding reported in a random sample of randomised, placebo controlled trials. BMJ 2004;328:432.doi:10.1136/bmj.328.74327.37952.631667.EE pmid:http://www.ncbi.nlm.nih.gov/pubmed/14761905
      OpenUrlAbstract/FREE Full Text
    92. ↵
      2. Sackett DL
      . Turning a blind eye: why we don't test for blindness at the end of our trials. BMJ 2004;328:1136.2.doi:10.1136/bmj.328.7448.1136-a pmid:http://www.ncbi.nlm.nih.gov/pubmed/15130997
      OpenUrlPubMed
    93. ↵
      2. Boutron I ,
      3. Estellat C ,
      4. Ravaud P
      . A review of blinding in randomized controlled trials found results inconsistent and questionable. J Clin Epidemiol 2005;58:1220–6.doi:10.1016/j.jclinepi.2005.04.006 pmid:http://www.ncbi.nlm.nih.gov/pubmed/16291465
      OpenUrlCrossRefPubMedWeb of Science
    94. ↵
      2. Hrobjartsson A ,
      3. Forfang E ,
      4. Haahr M
      . Testing the success of blinding in randomised clinical trials. Bridging the gaps. Abstracts of the 12th Cochrane Colloquium; 2–6 Oct 2004, Ottawa, Canada, 2004.
    95. ↵
      2. Boutron I ,
      3. Guittet L ,
      4. Estellat C , et al
      . Reporting methods of blinding in randomized trials assessing nonpharmacological treatments. PLoS Med 2007;4:e61. doi:10.1371/journal.pmed.0040061 pmid:http://www.ncbi.nlm.nih.gov/pubmed/17311468
      OpenUrlCrossRefPubMed
    96. ↵
      2. Probst P ,
      3. Zaschke S ,
      4. Heger P , et al
      . Evidence-based recommendations for blinding in surgical trials. Langenbecks Arch Surg 2019;404:273–84.doi:10.1007/s00423-019-01761-6 pmid:http://www.ncbi.nlm.nih.gov/pubmed/30824993
      OpenUrlCrossRefPubMed
    97. ↵
      2. Probst P ,
      3. Grummich K ,
      4. Harnoss JC , et al
      . Placebo-controlled trials in surgery: a systematic review and meta-analysis. Medicine 2016;95:e3516. doi:10.1097/MD.0000000000003516 pmid:http://www.ncbi.nlm.nih.gov/pubmed/27124060
      OpenUrlPubMed
    98. ↵
      2. Thybo KH ,
      3. Hägi-Pedersen D ,
      4. Dahl JB , et al
      . Effect of combination of paracetamol (acetaminophen) and ibuprofen vs either alone on patient-controlled morphine consumption in the first 24 hours after total hip arthroplasty: the PANSAID randomized clinical trial. JAMA 2019;321:562–71.doi:10.1001/jama.2018.22039 pmid:http://www.ncbi.nlm.nih.gov/pubmed/30747964
      OpenUrlPubMed
    99. ↵
      2. Perner A ,
      3. Haase N ,
      4. Guttormsen AB , et al
      . Hydroxyethyl starch 130/0.42 versus Ringer's acetate in severe sepsis. N Engl J Med 2012;367:124–34.doi:10.1056/NEJMoa1204242 pmid:http://www.ncbi.nlm.nih.gov/pubmed/22738085
      OpenUrlCrossRefPubMedWeb of Science
    100. ↵
      2. Schulz KF ,
      3. Altman DG ,
      4. Moher D , et al
      . CONSORT 2010 changes and testing blindness in RCTs. Lancet 2010;375:1144–6.doi:10.1016/S0140-6736(10)60413-8 pmid:http://www.ncbi.nlm.nih.gov/pubmed/20338625
      OpenUrlCrossRefPubMedWeb of Science
    101. ↵
      2. Kolahi J ,
      3. Bang H ,
      4. Park J
      . Towards a proposal for assessment of blinding success in clinical trials: up-to-date review. Community Dent Oral Epidemiol 2009;37:477–84.doi:10.1111/j.1600-0528.2009.00494.x pmid:http://www.ncbi.nlm.nih.gov/pubmed/19758415
      OpenUrlCrossRefPubMedWeb of Science
    102. ↵
      2. Sackett DL
      . Commentary: Measuring the success of blinding in RCTs: don't, must, can't or needn't? Int J Epidemiol 2007;36:664–5.doi:10.1093/ije/dym088 pmid:http://www.ncbi.nlm.nih.gov/pubmed/17675306
      OpenUrlCrossRefPubMedWeb of Science
    103. ↵
      2. Kirsch I
      . Response expectancy as a determinant of experience and behavior. Am Psychol 1985;40:1189–202.doi:10.1037/0003-066X.40.11.1189
      OpenUrlCrossRefWeb of Science
    104. ↵
      2. Kirsch I
      . Response expectancy theory and application: a decennial review. Applied and Preventive Psychology 1997;6:69–79.doi:10.1016/S0962-1849(05)80012-5
      OpenUrl
    105. ↵
      2. Kirsch I ,
      3. Wampold B ,
      4. Kelley JM
      . Controlling for the placebo effect in psychotherapy: noble quest or tilting at windmills? Psychol Conscious Theory Res Pract 2016;3:121–31.doi:10.1037/cns0000065
      OpenUrl
    106. ↵
      2. Boutron I ,
      3. Estellat C ,
      4. Guittet L , et al
      . Methods of blinding in reports of randomized controlled trials assessing pharmacologic treatments: a systematic review. PLoS Med 2006;3:e425. doi:10.1371/journal.pmed.0030425
    107. ↵
      2. Rabkin JG ,
      3. Markowitz JS ,
      4. Stewart J , et al
      . How blind is blind? Assessment of patient and doctor medication guesses in a placebo-controlled trial of imipramine and phenelzine. Psychiatry Res 1986;19:75–86.doi:10.1016/0165-1781(86)90094-6 pmid:http://www.ncbi.nlm.nih.gov/pubmed/3538107
      OpenUrlCrossRefPubMedWeb of Science
    108. ↵
      2. Margraf J ,
      3. Ehlers A ,
      4. Roth WT , et al
      . How "blind" are double-blind studies? J Consult Clin Psychol 1991;59:184–7.doi:10.1037/0022-006X.59.1.184
      OpenUrlCrossRefPubMedWeb of Science
    109. ↵
      2. Nielsen EE ,
      3. Nørskov AK ,
      4. Lange T , et al
      . Assessing assumptions for statistical analyses in randomised clinical trials. BMJ Evid Based Med 2019;24:185–9.doi:10.1136/bmjebm-2019-111174 pmid:http://www.ncbi.nlm.nih.gov/pubmed/30948454
      OpenUrlAbstract/FREE Full Text
    110. ↵
      2. Gøtzsche PC
      . Blinding during data analysis and writing of manuscripts. Control Clin Trials 1996;17:285–90.doi:10.1016/0197-2456(95)00263-4 pmid:http://www.ncbi.nlm.nih.gov/pubmed/8889343
      OpenUrlCrossRefPubMedWeb of Science
    111. ↵
      2. Järvinen TLN ,
      3. Sihvonen R ,
      4. Bhandari M , et al
      . Blinded interpretation of study results can feasibly and effectively diminish interpretation bias. J Clin Epidemiol 2014;67:769–72.doi:10.1016/j.jclinepi.2013.11.011 pmid:http://www.ncbi.nlm.nih.gov/pubmed/24560088
      OpenUrlCrossRefPubMed

    Footnotes

    • Contributors SJ, CG, and JCJ initiated the study and wrote the first manuscript draft. SJ and FWF independently retrieved relevant trial reports from the selected databases, performed risk of bias assessments, and statistical analyses with ongoing supervision from CG, SS, IK, and JCJ. All authors read, commented on and approved the final manuscript.

    • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

    • Competing interests SS, and FWF receives salaried from the Mental Health Services in the Capital Region of Denmark. SJ, JCJ, and CG receives salaries from the Copenhagen Trial Unit, Centre for Clinical Intervention Research.

    • Provenance and peer review Not commissioned; externally peer reviewed.