Statistical errors

R Nuzzo - Nature, 2014 - nature.com
Nature, 2014nature.com
150| NATURE| VOL 506| 13 FEBRUARY 2014© 2014 Macmillan Publishers Limited. All
rights reserved old-fashioned sense: worthy of a second look. The idea was to run an
experiment, then see if the results were consistent with what random chance might produce.
Researchers would first set up a 'null hypothesis' that they wanted to disprove, such as there
being no correlation or no difference between two groups. Next, they would play the devil's
advocate and, assuming that this null hypothesis was in fact true, calculate the chances of …
150| NATURE| VOL 506| 13 FEBRUARY 2014© 2014 Macmillan Publishers Limited. All rights reserved old-fashioned sense: worthy of a second look. The idea was to run an experiment, then see if the results were consistent with what random chance might produce. Researchers would first set up a ‘null hypothesis’ that they wanted to disprove, such as there being no correlation or no difference between two groups. Next, they would play the devil’s advocate and, assuming that this null hypothesis was in fact true, calculate the chances of getting results at least as extreme as what was actually observed. This probability was the P value. The smaller it was, suggested Fisher, the greater the likelihood that the straw-man null hypothesis was false. For all the P value’s apparent precision, Fisher intended it to be just one part of a fluid, non-numerical process that blended data and background knowledge to lead to scientific conclusions. But it soon got swept into a movement to make evidence-based decisionmaking as rigorous and objective as possible. This movement was spearheaded in the late 1920s by Fisher’s bitter rivals, Polish mathematician Jerzy Neyman and UK statistician Egon Pearson, who introduced an alternative framework for data analysis that included statistical power, false positives, false negatives and many other concepts now familiar from introductory statistics classes. They pointedly left out the P value.
But while the rivals feuded—Neyman called some of Fisher’s work mathematically “worse than useless”; Fisher called Neyman’s approach “childish” and “horrifying [for] intellectual freedom in the west”—other researchers lost patience and began to write statistics manuals for working scientists. And because
nature.com