Regression and Hiring a New Coach

The Blue Jackets

A coach I don't like.

The Columbus Blue Jackets hired a coach I don't like. I don't really care about the team, but I dislike the coach a lot -- he is a known psycho and he did a bad job with the Canucks. I also dislike the pattern that in the NHL, an unsuccessful coach will always get re-hired after a string of failures -- we can say that NHL culture respects a coach's experience far more than they respect his talent or success.

So I think the Blue Jackets made a terrible move and hired an idiot, but I also think they'll start winning a lot more games now.

The Blue Jackets are 0-7 this season, and they have a goal differential of -21. That's the worst in the league. Basically any level of performance, over their next 75 games, will be a huge improvement! Some people will see this improvement and say their new coach must be doing a great job.

The Blue Jackets' goalie has been saving 83.5% of the shots he faced this year, compared to his career average of 91.7%. Maybe he has actually gotten worse over the summer, but he probably hasn't gotten that much worse. So even if he bounces back to a weak average like 90% (or if they switch to a back-up goalie who can save 90%), the Blue Jackets will see a massive improvement in their goaltending. They will win more games. People might say that the team turned around and that their coach is doing a great job.

Regression to the Mean

The real cause of the team's improvement will be regression to the mean. Any ridiculous outcome, like the Blue Jackets' terrible start, is unlikely to persist. Ridiculous outcomes take crazy luck to occur in the first place, so it would be quite a coincidence for that luck to repeat itself for an entire year.

This cool infographic shows that a coaching change usually brings an improvement to a team's win percentage.

- Is this because the new coach must be more talented? Not really. In the NHL, most "new coaches" are just coaches that were previously fired by another team -- so comparing new and old coaches is really comparing the same people against themselves.

- Is this because the new coach marks a time for change and a fresh start? Possibly. When a team changes to a new coach, it is a chance to gather confidence and try some new strategies.

- Is this because of regression to the mean? I think so. When a hockey team fires its coach, the team is usually at their lowest point -- so improvement would have been likely to happen anyway. A bad NHL team will still usually win 40% of its games. So any time a team loses 7 games in a row, they're very unlikely to also lose their next 7 games in a row. A more likely result would be a 3-4 record in their next 7 games, which will be a great improvement.

The blue dots are above and below the line,
but the line is still the real pattern.

Hockey writers do talk about regression to the mean already, and usually they say it about teams that are over-achieving. When the Calgary Flames won a lot last year, everyone knew that they still weren't the best team and their results were fueled by luck. Then, people criticized the team by talking about "regression" -- they knew the Flames performance would decline to their skill level sooner or later.

The word "regression" sounds negative, so in hockey, it usually means a team will stop getting lucky wins. But in math, "regression" just means that the pattern of luck will go away soon -- extreme outcomes will cancel out, and looking at long-term data will show the "real" pattern in the results. Or instead, "regression to the mean" says that extreme outcomes are unlikely to repeat themselves, so when you do the next test, the results should drift back ("regress") towards the average (the "mean"). In hockey, this principle would say that an unlucky team at 0-7 will finally start getting some occasional wins.

Regression and Timing

Regression, then, means an inevitable end to either good luck and bad luck. Now matter how crazy a team's start was, regression will return them closer to the ordinary result of winning 50% of their games.

In Thinking Fast and Slow, Daniel Kahneman introduces the concept of regression with a story of an instructor who trains flight cadets. The instructor has observed that, when a cadet screws up a practice flight, he can scream at the cadet and then the cadet's performance will improve on the next flight. The instructor is confident that the screaming is what causes the improvement. Kahneman points out that after a bad flight, the next flight is very likely to be an improvement, regardless of whether anyone screamed at anyone.

The Blue Jackets' new coach is a man who is famous for "improving" bad teams by screaming at them. A month from now, people will write articles praising this coach for team's improvement and saying how smart it was to scream at the Blue Jackets. But the Blue Jackets could improve their current performance by hiring a dog as their coach. They could even have improved their performance by saving money and keeping their old coach.

A new coach can arrive when a team is getting bad results and then claim credit when a team starts getting average results. The coach will show up at the right time and appear to have "fixed some of the problems," but we know that regression would have fixed these problems anyway. The improved results should be no indication of improved performance.

An Attempt at Fairness

It is unfair that I write this about a coach I dislike, when the same criticism could be used to discredit any coach. So, I will give this new coach a test -- he must get the Blue Jackets to win 45% of their games. An average team should be able to win 50% (usually enough to make the playoffs), and a bad team should plan on winning 40%. The Blue Jackets are a bad NHL team, but they still won more than 50% last year and the year before. So 45% is a reasonable target for the rest of their season.

Since there are 75 games left, the new coach must go 34-41 to meet my challenge. This will leave his team out of the playoffs, but it will show more improvement than if they had just regressed to a 40% bad-team standard. If the Blue Jackets win 34 games this year, then we'll say their new coach has done a decent job. (It might be more telling to look at the Blue Jackets' goal differential or shot differential, but I'll frame this as a matter of wins for simplicity.)

A more sensible way to measure a coach's effect is to throw out the losing streak that led into their hiring, or to compare a team's win-percentage before for very long runs -- perhaps a full 82 games before and after the coaching change. Another casual way to measure a coach's effectiveness is to see how long they usually last with a team before hitting a bad streak and getting fired again.

Regression and Therapy

I wonder about regression in my own life. I started working with a new therapist this summer, and I noticed right away that my habits were improving and that I was not having as many extreme thoughts. I like my therapist and I think I'm receiving effective treatment, but I should be very suspicious. After all, I switched therapists when I was at my lowest point -- so it's possible that I was already likely, sooner or later, to bounce back to being my normal self. I experienced a nice recovery while working with my new therapist, but how do I know I couldn't have achieved the same recovery by hiring a dog as my therapist?

I spend a lot of money on therapy, and the Blue Jackets spent a lot of money to hire their new coach. Success is important to us, and we are assuming that this success comes from working with experts, not just from seeing our luck turn around. My goal in therapy is to get better and healthier than my usual long-term self. Getting better and healthier than my lowest point is something that probably would have happened anyway.