Is it time to send the Michigan Consumer Sentiment Survey off to the not-so-useful data junk heap?

The answer to this question hinges on whether the survey is really useful or not. Useful data will quantify information that tells us something useful about what it purports to measure. In the case of the University of Michigan's Consumer Sentiment Survey, it should provide accurate information about how a sampling of American consumers surveyed by University of Michigan academics view the state of the U.S. economy at the time it was taken.

For the surveyed results to qualify as useful information, which is to say information that policy makers can use to make sound decisions in setting policies, the surveyed sample must be representative of the U.S. population as a whole. Because if the sampling doesn't meet that statistical requirement, any policy set according to the survey's results would at a high risk of failure because it would directly lead to false conclusions. It's the policy wonk version of the "Garbage In, Garbage Out" problem from computer programming.

From the outside looking in, we should be able to tell whether a survey provides useful information from information about the population sample used to compile its findings. If the characteristics of the sampling reliably matches the characteristics of the U.S. population, then the survey is probably giving a good reading on how consumers see the U.S. economy. But if it deviates too far from the characteristics of the general population, then the odds that the survey is outputting results that belong in the garbage go way up.

Sampling appears to be a big problem with the Michigan Consumer Sentiment Survey. Nate Silver recengly honed in on the Michigan academics' survey samples in recent years to explain why their work product has become less than useful data:

For nearly four years, the internet has debated whether we’ve been mired in what Kyla Scanlon dubbed a “vibecession” — whether people feel worse about the economy than the underlying data suggests they “should” feel. People as esteemed as Nobel laureate Paul Krugman have frequently posted about the vibes mystery. Nate wrote a whole piece about the divergence in the New York Times two years ago.

But there’s one big problem with the discussion: most of the participants are relying on a broken survey, the University of Michigan’s consumer sentiment survey (“Index of Consumer Sentiment” or “ICS”), that is in dire need of being repaired. Failure to correct for these issues has led to plenty of pet theories — but they explain a trend that may not even exist.

It wasn't always that way. Silver notes the degradation of the quality of the Unviersity of Michigan survey is a recent development:

The University of Michigan ICS is the gold standard sentiment survey measuring consumer sentiment. The survey has historically shown a very strong correlation with “hard” economic data such as inflation and unemployment. But before more bad analysis gets done on the vibecession, people need to know they’re working with dubious data. As with election polls, the ICS has struggled amid a shift away from telephone polling. There are issues both with partisan nonresponse, with some political groups more likely to respond than others, and partisan expressive response, with survey-takers using questions about the economy to express political sentiment.

So the problems with the ICS are these:

1. The switch to online polling made responses more negative and,
2. There are too many Democrats in the sample.

Thus, ICS data since mid-2024 is not comparable to past periods.

Silver goes into far more detail in his analysis, we do recommend reading the whole thing. Here's his conclusion:

The conclusion is simple: the ICS cannot continue to ignore its sample’s skewed partisanship in the future. And people who write about consumer and voter sentiment shouldn’t ignore the problems either. President Trump is highly unpopular, and, in contrast to his first term, his economic numbers are worse than his overall ratings. But the Michigan survey exaggerates just how sour consumers are feeling about the economy – the vibecession is partly an artifact of bad data.

The changes in the University of Michigan's Consumer Sentiment Survey methodology and the excessively politicized slant of its recently targeted samples of the U.S. population have impaired the reliability of the survey. In effect, the survey's previous "gold standard" status has been debased. The Michigan Consumer Sentiment Survey has become less than useful data. Worse, there is no evidence as yet the academics who manage it are attempting any course correction to make its data useful for drawing valid conclusions.

Previously on Political Calculations

Here are all the articles in the "Less Than Useful Data" series!

• Less Than Useful Data in the World of Big Data
• Less Than Useful Data: Consumer Confidence?
• Less Than Useful Data: FHFA House Price Index
• Less Than Useful Data: Index of Leading Economic Indicators
• Less Than Useful Data: Weekly Chain Store Sales
• Less Than Useful Data: Challenger Layoff Announcements
• Less Than Useful Data: ADP Employment Index
• Less Than Useful Data: Inflation Indices
• Less Than Useful Data: The Real-World Costs of Bad Data
• Less Than Useful Data: Michigan Consumer Sentiment Survey

Image credit: Reliability and Validity by Nevit Dilmen on Wikimedia Commons Creative Commons CC BY-SA 3.0 Attribution-Share Alike 3.0 Unported Deed.