In March 2016 the American Statistical Association (ASA) released a formal statement regarding p-values. The statement addresses concerns that statisticians have warned about for decades, and should have a profound impact on the way research is conducted. It addresses topics that researchers and statisticians alike should be aware of, and ought to be taken into account in academic consulting. Below are six principles, followed by three excerpts found in the statement.
- P-values can indicate how incompatible the data are with a specified statistical model.
- P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.
- Scientific conclusions and business or policy decisions should not be based only on whether a p-values passes a specific threshold.
- Proper inference requires full reporting and transparency.
- A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.
- By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis.
"Practices that reduce data analysis or scientific inference to mechanical, 'bright-line' rules (such as 'p < 0.05') for justifying scientific claims or conclusions can lead to erroneous beliefs and poor decision making."
"The widespread use of 'statistical significance' (generally interpreted as 'p < 0.05') as a license for making a claim of a scientific finding (or implied truth) leads to considerable distortion of the scientific process."
"Cherry-picking promising findings, also known by such terms as data dredging, significance chasing, significance question, selective inference and 'p-hacking', leads to a spurious excess of statistically significant results in the published literature and should be vigorously avoided."
http://amstat.tandfonline.com/doi/abs/10.1080/00031305.2016.1154108