What it means when scientists say their results are 'significant'

What it means when scientists say their results are 'significant' At times statistical analysis can suggest a result is significant, but it may mean very little.

This article is part of the This is research series published by The Conversation where academics were asked to share and discuss open access articles that reveal important aspects of science. Today’s piece is authored by Yazad Irania post-doctoral researcher at South Australian Health & Medical Research Institute who looks at statistics and shares how to interpret them for meaning in the real world.

Let’s face it, scientific papers aren’t exactly page turners. They are written by scientists, for scientists, and often in a language that seems to only vaguely resemble English.

And perhaps one of the most daunting aspects of a scientific paper is the statistics (“stats”) section.

But what do stats really mean in the real world? Here’s an example from leukaemia research to help you break it down.

Read more: Our survey found 'questionable research practices' by ecologists and biologists – here's what that means


Strength of results

Stats are key to good research – they help researchers determine whether the results observed are strong enough to be due to an important scientific phenomenon.

As a research student I would always look for the magic number which indicates statistically significant differences in my experiments: most people agree this number to be 0.05 (you may see this in a paper written as p < 0.05).

When comparing two groups in a scientific study, statistical significance indicated by a p-value of less than 0.05 means that, in the case where there was no real difference between groups, there’s less than a 5 per cent chance of the observed result arising.

But the focus on looking for statistically significant differences can blind us to the bigger picture. As I advanced through my scientific training I learnt to look for biologically significant differences.


Read more: The curious case of the missing workplace teaspoons


Biological significance

Biological significance addresses the question of whether the statistical difference actually means anything in terms of a real outcome, like a disease. Can the result explain how the disease is caused? Does it provide a new avenue to treat the disease? Basically, is it relevant?

A recent paper published in the journal Leukaemia will help explain my point. The paper looked at why some people are able to stop their treatment for chronic myeloid leukaemia without the cancer coming back, while in others the cancer relapsed.

The key finding of this study was that patients who did not relapse had a higher proportion of natural killer cells compared to patients that did relapse.

Natural killer cells are a type of immune cell that controls viral infections and tumours. So, the more cells there are to kill the cancer, the less likely the cancer was to relapse – makes sense!

This finding has the potential to guide doctors in seeing which patients are likely to remain cancer-free after stopping treatment. This is definitely biologically significant.


Read more: My cancer is in remission – does this mean I'm cured?


Not so relevant

Another result from the same paper (Figure 3a if you want to click through to the data) shows a statistically significant difference in a sub-type of natural killer cells (called adaptive natural killer cells). But is this difference biologically relevant?

At this stage there is little evidence of a role for adaptive natural killer cells in the context of leukaemia. Also, the difference between the groups is relatively small, with a large variation within the groups (there are large error bars on the graph).

These factors make it more likely that the differences may be due to the mathematics involved in the statistical test rather than a biological effect. As with any new finding, time and further studies will be vital in working out whether this result actually means anything.

Act like an expert

So how do you pick if the statistical differences have biological value? Being a highly trained expert in the field certainly helps.

Another way to determine if the findings in a paper have biological relevance is to look for other papers that show similar results. If a result is “real” it should be found by other scientists who will build on it and publish more papers.

This means there will be lots of papers for you to read and apply your new-found passion for statistics.


The open access research paper for this analysis is Increased proportion of mature NK cells is associated with successful imatinib discontinuation in chronic myeloid leukemia.


The definition of statistical significance has been edited since this article was first published.The ConversationThis article is republished from The Conversation under a Creative Commons license. Read the original article. 

 
Donate to research and help find more effective treatments for kids with cancer.