Statistical vs. Managerial Significance
As market researchers, it is our task to sort the wheat from the chaff. Whether the data is from a survey or customer transactions it is our function to extrapolate meaning and educate those who can do something with it. When we dig into the details underlying our data we are often confronted with a fundamental question:
What is significant and what is not?
Statistical significance has a long history. There have been many great statisticians, economists and applied mathematicians who have contributed to the body of knowledge that we as market researchers use daily. There are several tests and procedures from which to choose depending upon our hypotheses and the complexity of the data. Survey researchers commonly employ a chi-square, t-test, F-test, or many others with our banner tables, ANOVAs, factor analyses, and regressions. Yet the question remains what is significant and what is not?
When you are blessed to have a significant number of responses to your survey then you can easily run into this question head on. A recent example involves comparing responses from two annual employee training surveys, with a combined 12,500 respondents. When examining the proportion that trained on a particular technology I noticed a three percentage point difference between the two iterations. Under normal circumstances (aka those with far fewer responses) I would dismiss such a difference as not significant, but the impact of the large respondent pool forced my hand. The difference was significant at the .001 level, certainly more so than the gold standard of 0.05.
Hence the dilemma – does statistically significant mean the same thing as managerially significant? The answer to this question calls upon the judgment of the researcher. In my reporting, I made the decision to report the difference as statistically significant. What would make it managerially significant? One has to look to the bottom-line impacts before making such a call. If a three percent drop in your customer base or sales volume translates to a substantial drop in profitability then the figure has both statistical and managerial significance.
As researchers, we are tasked with taking a holistic view of the data as it represents the market. Not all of the findings from our research will have a measurable impact on the business, at least in the short-run. The essential guidance then is to keep your reporting focus on the most salient points and reserve those that are off-center for ‘color commentary’.
