Confidence Intervals

Statisticians stress the importance of using confidence intervals (CIs). There is, however, debate over which type of CIs to use and how to best define and interpret them. In spite of this confusion, you should use CIs to express the results of statistical tests because they convey more information than P values alone.

StatsDirect documentation uses the common (see below) interpretation of CIs. The CI included with each StatsDirect function is discussed in the help text for that function. In order to understand how CIs relate to specific statistical methods, read the interpretation of CI in the worked examples of StatsDirect help text.

The confidence level sets the boundaries of a confidence interval, this is conventionally set at 95% to coincide with the 5% convention of statistical significance in hypothesis testing. In some studies wider (e.g. 90%) or narrower (e.g. 99%) confidence intervals will be required. This rather depends upon the nature of your study. You should consult a statistician before using CI's other than 95%.

You will hear the terms confidence interval and confidence limit used. The confidence interval is the range Q-X to Q+Y where Q is the value that is central to the study question, Q-X is the lower confidence limit and Q+Y is the upper confidence limit.

Familiarise yourself with alternative CI interpretations:

Common

A 95% CI is the interval that you are 95% certain contains the true population value as it might be estimated from a much larger study.

The value in question can be a mean, difference between two means, a proportion etc. The CI is usually, but not necessarily, symmetrical about this value.

Pure Bayesian

The Bayesian concept of a credible interval is sometimes put forward as a more practical concept than the confidence interval. For a 95% credible interval, the value of interest (e.g. size of treatment effect) lies with a 95% probability in the interval. This interval is then open to subjective moulding of interpretation. Furthermore, the credible interval can only correspond exactly to the confidence interval if prior probability is so called "uninformative".

Pure frequentist

Most pure frequentists say that it is not possible to make probability statements, such CI interpretation, about the study values of interest in hypothesis tests.

Neymanian

A 95% CI is the interval which will contain the true value on 95% of occasions if a study were repeated many times using samples from the same population.

Neyman originated the concept of CI as follows: If we test a large number of different null hypotheses at one critical level, say 5%, then we can collect all of the rejected null hypotheses into one set. This set usually forms a continuous interval that can be derived mathematically and Neyman described the limits of this set as confidence limits that bound a confidence interval. If the critical level (probability of incorrectly rejecting the null hypothesis) is 5% then the interval is 95%. Any values of the treatment effect that lie outside the confidence interval are regarded as "unreasonable" in terms of hypothesis testing at the critical level.