Originally published on April 29, 2009.
I’m a stats geek. I can’t help it. I did really well in my college stats class and enough of it has stuck to keep me slightly armed, dangerous, and highly skeptical.
Whenever I see a study about “X% of Y group doing something” I cringe. The group doing the study is rarely sited (The 11th grade class at the school down the block isn’t quite as reputable as the New England Journal of Medicine). The sample size is rarely mentioned (“my brother, his girlfriend and her sister” isn’t quite as reliable as a 5000 person sample). Base rates are never mentioned. (For example, if there’s a 0.05% chance of something occurring, then a 300% increase means there’s now a still highly unlikely 0.15% chance of that thing occurring. This is quite a bit different than starting with a base rate of 10%…a 300% increase now means a 30% chance of that thing occurring. A 0.15% chance vs. a 30% chance? But you wouldn’t know the difference if you’re only told about the 300% increase without reference to the base rate.)
I know, I know, stats on a wine blog, no fun. but like I said, I can’t help it. Misleading usage of statistics makes me a little bit crazy.
So I was a little bit crazed at yesterday’s Vinexpo event reviewing the findings of their recent global study on women and their wine buying habits. The study surveyed about 4,000 women across the US, UK, Japan, and Europe.
In each country, they partnered with one or two publications in order to identify respondents. For example, in the US, they partnered with the Wine Spectator, so the vast majority of US respondents were Wine Spectator subscribers. Not surprisingly, the vast majority of respondents drank wine at least once a week. If your sample universe is subscribing to one of the biggest wine publications in the US, you would expect them to be drinking a fair amount of wine. Does this mean that most women in the US drink wine at least once a week? No, not at all. From this survey, you can’t say anything about the larger population because the Wine Spectator readership isn’t representative of the larger population. To try to extrapolate (fancy stats term, still can’t help it) that way is misleading.
But still, press releases are issued and the study gets picked up and the results reported as if they were applicable to the general population. As in this piece in the UK Telegraph. And this piece in Harpers Wine & Spirits. Vinexpo’s sample in the UK was drawn from Decanter and Living magazines, which may be perfectly aligned with the demographics of the broader UK population….but probably aren’t.
So you read these stories and you think you understand women’s wine buying habits. But you don’t. You understand wine buying habits of Decanter readers and Living readers….except that difference is never stated, or even alluded to. But all those numbers followed by %-signs look very official, no?
So statistically speaking, this study, to the extent that it’s meant to reveal wine buying habits among the general female population, is flawed. As a qualitative study, or even a quantitative study among a very specific population subset, it’s fine. But it’s being reported as the former, without any mention of the flaws….and that’s what drives me crazy – the annoyingly incorrect application of statistics!
I know, I can’t help it, I’m a stats geek.