Sunday, May 17, 2015

Clear Comparisons Make Your Data Meaningful


Comparisons are crucial. They fill Abstracts, Results and Discussions: comparisons between groups, between theory and observation, between your study and others, and so on. With well-written comparisons, you demonstrate the clarity and logic of your analysis, and provide meaningful information; without good comparisons, your data lack meaning.

The example of an unclear comparison that I'm going to use comes from the abstract of a published, peer-reviewed paper. Because I don't want to embarrass the author, I've changed or omitted certain details, including the species, so this example does not present a real scientific finding. However, I have kept the structure of the comparison the same. 

An analysis of hematological indicators in puppies of breeds X and Y

Hematological indicators A, B, and C are used to determine the overall health status of dogs and to diagnose many diseases. The normal ranges of these indicators differ with age and sex, and recent studies also indicate that, in healthy adults of breeds X and Y, the values of these indicators can be outside what were previously considered to be the normal ranges for all dogs. However, it is not known if the levels of hematological indicators A, B, and C in healthy puppies of breeds X and Y can be outside the established norms. To answer this question, [...samples were taken from certain groups at certain times, etc.]. We found that, especially during the 2nd and 3rd months of life, the mean level of hematological indicator A was significantly higher in breed X (P < 0.05). [more details, which do not explain this comparison] 

Problem 1:
Higher than what? Higher than the established norms, than at birth, than in breed Y? The unknown and the research question suggest that the writer means "higher than the established norms", but we need to be sure. 

Solution:
Write complete comparisons: not "this was higher," but "this was higher than that." 

Problem 2: 
How much higher? Let's assume that the author is using "significantly" in the correct scientific sense, to indicate a "statistically significant" difference, not a "large" or "important" difference (Hofmann 2014). So we know that there probably is a real difference between the mean level of indicator A in breed X and whatever the author was comparing it to. But we don't know if this is a big or important difference. What if the difference is so small that it will be difficult to detect with the equipment that most veterinarians have? A "negative result" like this is still valuable, and the tendency not to publish negative results from medical studies is unscientific, unethical, and dangerous (Evans et al. 2011), but the result needs to be made clear. 

Solution: 
Remember, your readers need to know the size of the effect or the difference, not just that there probably is an effect or a difference (Evans et al. 2011, Nuzzo 2014). Be precise, and include information on both the size and the statistical significance, for example: 

"Although the difference was significant, the mean level of hematological indicator A was only slightly higher in breed X at 4 weeks of age than at 2 weeks of age (4 weeks [mean, standard deviation], 2 weeks [mean, standard deviation], P < 0.05)."  

Conclusion: 
Before you submit your paper, check it for these kinds of errors. It's easy to forget and write an incomplete or unclear comparison, and I've done it many times myself—in fact, a student found one in my writing just this Friday! 

References:  
Evans I, Thornton H, Chalmers I, Glaszou P. 2011. Testing Treatments: Better Research for Better Healthcare. Pinter & Martin, Ltd: London. p.88–89, 96–7, 163.
Hofmann, AH. 2014. Scientific Writing and Communication: Papers, Proposals, and Presentations. Oxford: Oxford University Press. p. 17, 270.
Nuzzo R. 2014. Statistical Errors: P values, the 'gold standard' of statistical validity, are not as reliable as many scientists assume. Nature 506:150-152.

No comments:

Post a Comment