Statistical formulas don’t know whether they are being used properly, and they don’t warn you when your results are incorrect. In order to draw the appropriate conclusions, it’s up to you to avoid overstating your results and you need to find accurate cause-and-effect relationships from your results.
Some of the most common mistakes made in conclusions are overstating the results or generalizing the results to a larger group than was actually represented by the study. For example, a professor wants to know which Super Bowl commercials viewers liked best. She gathers 100 students from her class on Super Bowl Sunday and asks them to rate each commercial as it is shown. A top-five list is formed, and she concludes that all Super Bowl viewers liked those five commercials the best. But she really only knows which ones her students liked best — her students are not a representative sample of the population and she didn’t study any other groups, so she can’t draw conclusions about all viewers.
One situation in which conclusions cross the line is when researchers find that two variables are related (through an analysis such as regression) and then automatically leap to the conclusion that those two variables have a cause-and-effect relationship.
For example, suppose a researcher conducted a health survey and found that people who took vitamin C every day reported having fewer colds than people who didn’t take vitamin C every day. Upon finding these results, she wrote a paper and gave a press release saying vitamin C prevents colds, using this data as evidence.
Now, while it may be true that vitamin C does prevent colds, this researcher’s study can’t claim that. Her study was observational, which means she didn’t control for any other factors that could be related to both vitamin C and colds. For example, people who take vitamin C every day may be more health conscious overall, washing their hands more often, exercising more, and eating better foods; all these behaviors may be helpful in reducing colds.
Until you do a controlled experiment, you can’t make a cause-and-effect conclusion based on relationships you find.