Exercises

The above is illustrated through the following model situations:

1. The following example is from the book Information Theory, Inference, and Learning Algorithms by David MacKay (page 464), which is available at http://www.inference.org.uk/itila/book.html. Consider the discrete random variable $X\colon S\to(\theta,\theta+1)$, where $\theta\in\mathbb{Z}$, with density $f_\theta(\theta)=f_\theta(\theta+1)=1/2$. Hence, $X$ takes on either the value $\theta$ or the value $\theta+1$ with probability $1/2$ for either value. Let $X_1,X_2$ be a sample of size $n=2$ from the distribution of $X$ and define $X_{\min}:=\min\{X_1,X_2\}$ and $X_{\max}:=\max\{X_1,X_2\}$. By considering the four possible outcomes for $(X_1,X_2)$, show that $[X_{\min},X_{\max}]$ is a $75\%$ confidence interval for $\theta$.
2. Suppose that, in the context of the above example, we measure $X_1=20$, $X_2=21$. The $75\%$ confidence interval is then $[X_{\min},X_{\max}]=[20,21]$. What is the probability that $\theta$ lies in this interval?
3. Suppose that we measure $X_1=X_2=20$ so that the $75\%$ confidence interval is $[X_{\min},X_{\max}]=[20,20]$. No other information about $\theta$ is available, so we could have $\theta=20$ or $\theta=19$. Confirm, using Bayes’s Theorem, the intuitive statement that each value of $\theta$ is equally likely, i.e., the probability that $\theta$ lies in the $75\%$ confidence interval is $50\%$.
4. A continuous version of this example, originally found here, is as follows: Let $Y$ follow a continuous uniform distribution on the interval $[-1/2,1/2]$ and define $X=\theta+Y$. let $X_1,X_2$ be a random sample of size $n=2$ from the distribution of $X$. Set $X_{\min}:=\min\{X_1,X_2\}$ and $X_{\max}:=\max\{X_1,X_2\}$. Show that $[X_{\min},X_{\max}]$ is a $50\%$ confidence interval for $\theta$. Suppose that a measurement yields values such that $x_{\max}-x_{\min}\ge 1/2$. Show that the probability that the interval $[x_{\min},x_{\max}]$ contains $\theta$ is $100\%$.
5. A more complicated example with many facets is discussed in the article by Morey et al. mentioned at the top of this page. Study Example 1 and prove that both the Confidence Procedure 1 (sampling distribution of the mean) and the Confidence Procedure 2 (non-parametric) give $50\%$ confidence intervals. Explain and re-derive the graphics B and D in Figure 2. Explain what the horizontal dotted lines represent. Would you conclude that either of the two confidence procedures is “better” than the other?

An Alternative Phrasing

Since (S2) above is what most people would like to hear, while (S1) sounds confusingly similar, often (S1) is phrased as follows:

• (S1’) A $100(1-\alpha)\%$ confidence interval for a parameter $\theta$ is an interval $[L_1,L_2]\subset\mathbb{R}$ generated by a procedure that in repeated sampling has at least a $100(1-\alpha)\%$ probability of containing the true value of $\theta$, for all possible values of $\theta$.

This definition, following Neyman’s orginal version from 1937, is taken from the article by Morey et al. mentioned above. The idea that, in principle, probability statements are true if a sampling procedure can be repeated many times is at the heart of frequentist statistics. In practice, of course, most of the time only a single sample is taken. (This is one point of criticism leveled at frequentist statistics.)

It is notable that Neyman does not ascribe any other properties to a given confidence interval, in particular whether it actually contains $\theta$. So how should one interpret a confidence interval with respect to the actual value of $\theta$? The pithy answer is “not at all,” as the dialogue below illustrates.