Gostudymock3

Type I and Type II Errors in Hypothesis Testing



Hypothesis testing, which is heavily tested on the CFA Level 1 exam, is the process of testing whether a certain statement or observation about a population parameter is statistically meaningful.

To do this we use a sample of data to draw inferences about the underlying population. This requires us to assume a certain level of risk that the inferences we make are inaccurate.

How do we represent this risk?

The level of significance (α) is the proportion of false alarms we’re willing to tolerate in our decision process. In other words, the level of significance represents how willing we are to be wrong. At a 5% level of significance we are willing to be wrong 5% of the time and so on.

A hypothesis test is always conducted for a particular level of significance. The lower the error we are willing to tolerate, the higher the critical value of our given test statistic will have to be and the harder it becomes to reject a null hypothesis. 

However, no matter what critical value we use for our test statistic there is always a chance that we get the wrong results.

The Two Types of Mistakes in Hypothesis Testing

Within hypothesis testing we face two risks of being wrong. One risk occurs when we reject a true null hypothesis. The second risk happens when we fail to reject a null hypothesis that is false. These are “mutually exclusive” errors, you can’t make both at the same time.

  • Type I error – Rejecting a null hypothesis that is actually true
  • Type II error – Failing to reject a null hypothesis that is false

We interpret the significance level as the probability of making a type I error. So a 5% significance level gives us a 5% chance of rejecting a true null hypothesis.

The distinction between Type I and Type II errors will be tested on the CFA Level 1 exam, so let’s run through a scenario to give you a couple of ways to remember it.

Is a portfolio manager performing well? 

Let's take a scenario where we are evaluating whether a portfolio manager is adding value to our investment process. 

In this case our null hypothesis is that the manager is NOT adding any value. Our alternative hypothesis is that the manager IS adding value:

H0: Manager is adding value

HA: Manager is not adding value

In this case, a Type 1 error would occur if we rejected a true null hypothesis, meaning we kept a bad manager because we determined they were adding value when in fact they were not.

A Type II error would occur if we failed to reject a true null hypothesis, meaning we fired a good manager because we falsely concluded that they were not adding value when in fact they were. 

Type I error is keeping a bad manager and a type II error is firing a good manager.

How to remember which error is which?

If you remember the two hypotheses you can remember which error is which in two ways:

The first is via the expression true null horn. Where the ho in horn represents H0 (the null hypothesis), and the ‘RN’ in the word = reject null.

The second method is my preferred one. It's just the phrase “Type 1 keep a pisser” which is a very English way of saying that you're keeping a bad manager. 

Level of Significance and the Relationship between Type 1 and 2 Errors

Depending on the scenario, you may encounter situations in which the risk of making a Type 1 error is much higher than making a Type 2 error and vice versa. In this instance we can adjust our level of significance to lower the risk.

For example a 0.01 or 1% level of significance has a lower risk of a Type I error than a 5% level of significance. The tradeoff of course is that we’re raising the barrier required to reject the null hypothesis, even if that hypothesis is false. By reducing the probability of keeping a bad manager we’re increasing our probability of firing a good one. It's worth noting that in general we think of Type I errors as having greater risk.

Recapping Type 1 and Type 2 Errors for the CFA Exam

Within hypothesis testing we face two risks of being wrong. One risk occurs when we reject a true null hypothesis. The second risk happens when we fail to reject a null hypothesis that is false. 

  • Type I error – Rejecting a null hypothesis that is actually true
  • Type II error – Failing to reject a null hypothesis that is false

It is usually worse to make a type 1 error. 

Type 1 and 2 Errors - CFA Exam