Multiplicity and Bonferroni corrections

Multiplicity issues exist when multiple null hypotheses are tested. Testing more than one null hypothesis increases the risk of getting at least one false-positive test above the nominal significance level, see Figure 1. The phenomenon is important to take into account in confirmatory studies, and one way to do this is to use a Bonferroni correction, i.e. by lowering the significance level by a factor of 1/m, where m is the number of tested null hypotheses.

Figure 1. The relation between the number of tested null hypotheses and the risk of at least one false-positive test.

However, to avoid subjectivity, the adjustment should be pre-specified, and as it has negative effects on the statistical power of the comparisons, it should also be accounted for in the sample size calculation, and this increases patient numbers and costs.

Multiplicity problems can often be avoided in the study design by a careful definition of endpoints or solved by using closed test procedures or more effective adjustment methods such as Holm’s or Hochberg’s methods. In addition, while multiplicity issues is a problem in confirmatory studies, it is not relevant in non-confirmatory studies such as exploratory or hypothesis-generating investigations.

Furthermore, the statistical analysis of observational studies needs to include validity considerations as selection and confounding bias cannot be prevented in the study design, which implies that detailed pre-specification is not practically possible. Moreover, the strategy, common in laboratory studies, of Bonferroni correcting for the number of exposure groups but ignoring that multiple endpoints are tested, does not provide a reasonable solution to the multiplicity problem.