In statistics, if you want to draw conclusions about a null hypothesis H0 (reject or fail to reject) based on a p-value, you need to set a predetermined cutoff point where only those p-values less than or equal to the cutoff will result in rejecting H0.
While 0.05 is a very popular cutoff value for rejecting H0, cutoff points and resulting decisions can vary — some people use stricter cutoffs, such as 0.01, requiring more evidence before rejecting H0, and others may have less strict cutoffs, such as 0.10, requiring less evidence.
If H0 is rejected (that is, the p-value is less than or equal to the predetermined significance level), the researcher can say she’s found a statistically significant result. A result is statistically significant if it’s too unlikely to have occurred by chance assuming H0 is true. If you get a statistically significant result, you have enough evidence to reject the claim, H0, and conclude that something different or new is in effect (that is, Ha).
The significance level can be thought of as the highest possible p-value that would reject H0 and declare the results statistically significant. Following are the general rules for making a decision about H0 based on a p-value:
If the p-value is less than or equal to your significance level, then it meets your requirements for having enough evidence against H0; you reject H0.
If the p-value is greater than your significance level, your data failed to show evidence beyond a reasonable doubt; you fail to reject H0.
However, if you plan to make decisions about H0 by comparing the p-value to your significance level, you must decide on your significance level ahead of time. It wouldn’t be fair to change your cutoff point after you’ve got a sneak peak at what’s happening in the data.
You may be wondering whether it’s okay to say “Accept H0” instead of “Fail to reject H0.” The answer is a big no. In a hypothesis test, you are not trying to show whether or not H0 is true (which accept implies) — indeed, if you knew whether H0 was true, you wouldn’t be doing the hypothesis test in the first place. You’re trying to show whether you have enough evidence to say H0 is false, based on your data. Either you have enough evidence to say it’s false (in which case you reject H0) or you don’t have enough evidence to say it’s false (in which case you fail to reject H0).
These guidelines help you make a decision (reject or fail to reject H0) based on a p-value when your significance level is 0.05:
If the p-value is less than 0.01 (very small), the results are considered highly statistically significant — reject H0.
If the p-value is between 0.05 and 0.01 (but not super-close to 0.05), the results are considered statistically significant — reject H0.
If the p-value is really close to 0.05 (like 0.051 or 0.049), the results should be considered marginally significant — the decision could go either way.
If the p-value is greater than (but not super-close to) 0.05, the results are considered non-significant — you fail to reject H0.