Making a Stat Less Significant

by Carl Bialik

Wall Street Journal
April 2, 2011

A group of mathematicians has been trying for years to have a core statistical concept debunked. Now, the Supreme Cout might have done it for them.

Last month, the court considered a case brought by investors in Matrixx Initiatives Inc. They alleged the company failed to disclose material information by neglecting to reveal it had received reports an over-the-counter medicine, Zicam Cold Remedy, caused a loss of sense of smell.

When those reports came to light, the company's stock fell. Eventually, the Food and Drug Administration warned consumers not to use certain Zicam products.

The company argued in court that the initial reports of the possible side effect didn't rise to the level of "statistical significance," and therefore didn't need to be disclosed.

Last week, the Court rejected that argument, ruling unanimously that the case against Matrixx pending in a lower court could proceed. In their opinion, the justices said companies can't only rely on statistical significance when deciding what they need to disclose to investors.

Amen, say several statisticians who have long argued that the concept of statistical significance has unjustly overtaken other barometers used to determine which experimental results are valid and warrant public distribution. "Statistical significance doesn't tell you everything about the truth of the hypothesis you're exploring," says Steven Goodman, an epidemiologist and biostatistician at the Johns Hopkins Bloomberg School of Public Health.

A point on which most statisticians agree is that statistical significance is difficult to explain.

More