Are Retailers and CPGs Getting Research Sampling Wrong?
The notion that consumer sampling is only as good as the sample seems straightforward enough, but new insights reveal that companies relying on shopper research may not be getting the whole or the right picture of their audience. According to a recent white paper from the 84.51° retail data science arm of The Kroger Co., there are some notable shortfalls in traditional sampling methodologies.
The paper is based on an 84.51° study of the quality and effectiveness of different consumer research formats. “We wanted to understand how sampling has an impact on research and research outcomes,” Nancy Reilly, VP of consumer research, told Progressive Grocer in a recent interview.
[Read more: “Why CPGs Are Looking Beyond Pricing”]
Reilly noted that the 84.51° team didn’t go into their study with a complete set of hypotheses and were surprised by some of the findings. For example, the study found that only 25% of people in traditional research recruiting methods should actually qualify for a study.
Why the yawning gap? The firm’s researchers determined that self-identified respondents often overstate or wrongly state key metrics. Using 84.51° and Kroger shopping data, the study showed that 60% of self-claimed respondents were misclassified into the wrong buyer groups and 75% were found to have zero purchases in the category they claimed to purchase. In addition, more than 10% of self-claimed respondents said they buy all categories, compared to the 1% of shoppers that actually do that, based on real transactional data.
“There are thing respondents don’t remember well, like ‘How much did you spend on this product?’ or ‘How many times did you buy it?'” Reilly pointed out.
If self-responding consumers often inflate or confuse their answers, other sampling methods give a clearer view of shopper sentiments and actions. The study affirmed that behaviorally-verified respondents who are identified based on actual purchase behavior from transaction data provide more accurate insights.
“We feel good about that form of behaviorally-verified human respondents. We know they shop in stores,” Reilly said, adding that advances in technology can muddy the waters. “In the research industry there are challenges with bots taking surveys, and what we are seeing with AI and Gen AI is that bots are getting smarter.”
While 84.51°’s study underlines the accuracy of behavior-based data, the findings also reflect the importance of managing the sampling process. “It’s making sure that when a retailer or brand wants to do research that they ask a lot of questions, like, ‘How do you do sampling?’ ‘What quality perimeter do you do around sampling?’” Reilly said, adding that those questions are pivotal because the resulting answers are used to make decisions. “At the end of the day, we want to talk to humans to understand their point of view and their needs.”
The full paper is available online.
