1️⃣ Misunderstanding What a P-Value Represents
Explanation:
A p-value does not tell you the probability that the null hypothesis is true. It simply tells you how likely your observed data would be, assuming the null hypothesis is correct.
Common Mistake Example:
“SINCE p = 0.04, there’s a 96% chance the treatment works.”
👉 Incorrect — p-value ≠ probability the hypothesis is true.
2️⃣ Confusing Statistical Significance with Clinical Importance
Explanation:
A small p-value (e.g., p < 0.05) doesn’t necessarily mean the result is clinically meaningful. The effect size and clinical relevance should always be considered.
Example:
A blood pressure drug shows a statistically significant reduction of 1 mm Hg (p < 0.001) — but that small change may not be clinically useful.
3️⃣ Incorrect Interpretation of Confidence Intervals (CIs)
Explanation:
A 95% CI doesn’t mean there’s a 95% probability that the true value lies in the interval. It means that if we repeated the study many times, 95% of those intervals would contain the true parameter.
Example Mistake:
“There is a 95% chance that the true odds ratio is between 1.2 and 2.0.”
👉 Incorrect — instead, say: “We are 95% confident that the true odds ratio lies within this range.”
4️⃣ Ignoring Confidence Intervals in Favor of P-Values Only
Explanation:
P-values give a binary yes/no idea of significance but CIs provide a range of plausible values and more insight into effect size and precision.
Example:
Two studies may both have p < 0.05, but one CI is narrow (precise estimate), and the other is very wide (uncertain estimate).
5️⃣ Dichotomizing P-Values (Significant vs. Non-Significant)
Explanation:
It is a mistake to treat p = 0.049 as “significant” and p = 0.051 as “non-significant.” P-values exist on a continuum — such small differences should not drive major conclusions.
Example:
A study with p = 0.051 still shows suggestive evidence and should be considered in context, not discarded as “negative.”
6️⃣ Failing to Adjust for Multiple Comparisons
Explanation:
When multiple hypotheses are tested, the chance of a false positive increases. P-values must be adjusted (e.g., using Bonferroni correction) when multiple comparisons are made.
Example:
Testing 20 outcomes with p < 0.05 will, by chance, likely produce 1 false-positive result — unless corrected.
