The benefits of sharing null results
This section explores positive attitudes towards sharing null results as well as any benefits that researchers have experienced after sharing null results.
Challenges in sharing and publishing results are explored in Section 3.
Most researchers (85%) say that it is ‘somewhat’ or ‘very important’ to share null results – although there could be a self-selection bias considering the topic of the survey that they chose to participate in (n=11,069).
1.1 Perceived benefits
98% of researchers recognise the benefits of sharing null results, with almost two thirds thinking that null results can inspire new hypotheses or methodologies; prevent the duplication of unnecessary research; and/or increase transparency in research. Over half think that sharing null results increases the likelihood of improving methodology and/or enables others to build on research.
Figure 1: Researchers’ perception of benefits in sharing null results
Table 1: Researchers’ perception of benefits in sharing null results
|
Why do you think null results should be shared, if at all? Please select all that apply. |
|
|---|---|
|
To inspire new hypotheses or methodologies in relation to the research topic |
65% |
|
To prevent the duplication of unnecessary research |
64% |
|
To increase transparency in research |
63% |
|
To increase the likelihood of improving methodology |
58% |
|
To enable others to build on your research |
56% |
|
To help identify false positives in research |
50% |
|
To improve the reproducibility of research |
46% |
|
To inform other researchers that it is not worth following this line of research |
45% |
|
To reduce any stigma that may be associated with null results |
37% |
|
Other reason |
5% |
|
I don’t know / Unsure |
0.2% |
|
Not applicable – I do not believe null results should be shared |
1% |
Sample: All responses (n=11,069); Unweighted.
When asked to what extent they agreed that ‘sharing null results improves the quality of subsequent research’, 88% agreed (n=11,069).
1.2 Experienced benefits
Researchers were asked about their experiences both as readers and as authors to understand the impact of sharing null results on researchers in their different roles throughout the research lifecycle.
1.2.1 Benefits experienced as a reader
68% recall utilising null results shared by others, with the top three benefits being: sharing null results better informed their hypothesis, led to them adapting their methodology, and/or inspired a piece of future research (n=11,069).
“When I came across research that produced null results, I adapted my planned methodology to avoid redundant work and potential failure. This helped me refine my hypothesis and identify alternative approaches before proceeding with an experiment, ultimately saving time and resources. Furthermore, these null results inspired me to reframe my understanding of the research question and sometimes shift my focus toward more promising avenues or theories.”
Figure 2: Benefits of reading or using null results shared by others
Table 2: Benefits of reading or using null results shared by others
|
We would now like you to think about the null results you may have learned of via your peers or the research |
|||
|---|---|---|---|
|
Better informed my hypothesis for related research |
39% |
68% |
net used |
|
Adapted my methodology for related research |
34% |
||
|
Inspired a piece of future research |
33% |
||
|
Adapted planned research to avoid duplicating the research that produced null results |
27% |
||
|
Cited in a journal or book publication |
22% |
||
|
Cancelled planned research to avoid duplicating the research that produced null results |
13% |
||
|
Cited in a grant proposal |
11% |
||
|
Read but did not use |
10% |
||
|
Never used |
13% |
||
|
I can’t remember |
8% |
Sample: All responses (n=11,069); Unweighted.
Respondents commented on the topic of how sharing null results can reduce research waste and duplication, for example:
“I was told a widely used methodology did not show what it was supposed to. Also, that antibodies from several major companies were poor and next to useless. But I only ever saw that information in 2 journal articles (briefly mentioned, years later) buried among other information in the articles. Had I known beforehand if it had been widely disseminated in my field, it would have saved me 2+ critical research years!”
1.2.2 Benefits to authors publishing null results in a journal
The survey made a distinction between sharing null results by any channel (see Section 2.3), and publishing null results in a journal, to understand whether journal publication had played any role in benefits experienced by authors.
From respondents who have submitted a manuscript based (solely or mostly) on null results to a journal, and whose most recent manuscript based on null results was accepted for publication (n=1,228), 72% reported a positive outcome from publishing null results in a journal. However, 20% shared concerns which were mostly about negative impacts on their reputation or career. Additionally, 16% responded that they have not experienced any outcomes as a result of publishing null results in a journal.
The top three positive experiences included: inspiring a new hypothesis or methodology in relation to the research topic (39%), help identifying issues with the methodology (29%), preventing the duplication of unnecessary research (28%).
Figure 3: Experiences of publishing null results in a journal
Table 3: Experiences of publishing null results in a journal
|
Which of the following, if any, did you experience as a result of publishing null results in a journal? Please select all that apply. |
|||
|---|---|---|---|
|
Inspired a new hypothesis or methodology in relation to the research topic |
39% |
72% |
net |
|
Helped identify issues with the methodology |
29% |
||
|
Prevented the duplication of unnecessary research |
28% |
||
|
Positive impact on my career |
20% |
||
|
Positive impact on my reputation in the field |
20% |
||
|
Inspired a collaboration with another research group |
18% |
||
|
Inspired another researcher who tried to reproduce the research |
17% |
||
|
Concerns I may be viewed negatively by my peers |
8% |
20% |
net |
|
Concerns regarding feeling embarrassed / like a failure |
7% |
||
|
Negative impact on my career |
6% |
||
|
My citation metrics (e.g. H-index) were negatively impacted |
5% |
||
|
Negative impact on my reputation in the field |
3% |
||
|
Not applicable – I have not experienced any outcomes as a result of publishing my null results in a journal |
16% |
||
|
I don’t know / Unsure |
9% |
Sample: All responses from population who have “Submitted a manuscript based (solely or mostly) on null results to a journal” and most recent manuscript was accepted for publication (n=1,228); Unweighted.
Looking at examples of these top three positive experiences for authors, publishing null results in a journal has:
Successfully challenged prior theory and changed the status quo in their field
“Since we published the article that has negative results, other researchers have cited the article. People are happy and it has changed the false information held for many decades. Note that the publication has some impact but very helpful for those in community medicine. During this period, we have been able to sensitize women in the rural areas [...]. That's one of importance of null result.”
“By sharing the null results informally with the journal, we found other groups with similar results (and who had previously been rejected) and collaborated with them to submit three back-to-back papers. Now, this forms the mainstream of research in this particular area. So, in fact, the publication transformed null results into positive results.”
Inspired a new or refined methodology, hypothesis, or line of research
“Publishing a null result has helped other groups adjust their hypothesis and experimental approach, and the community has advanced more as a consequence.”
“My results were not in line with those of my peers. Indeed, they opened a new research field.”
Limited research waste by preventing the duplication of unnecessary research
“The negative results paper we published was widely cited and revealed that others had in fact tried our type of experimentation and generated negative results as well. This allowed them to cite our publication to prevent others from spending time on unproductive methodologies.”
“Someone reached out to me telling it was good we published the study, because they had planned a similar study but cancelled it due to our null result.”
Further positive responses indicated impacting their career (20%) and on their reputation in the field (20%). 18% said that publishing null results inspired a collaboration with another research group, and 17% said that it inspired another researcher who tried to reproduce the research
Examples taken from the free-text comments show that publishing null results has:
Led to collaborations or networking opportunities
“Our null results paper provided evidence against one of two competing hypotheses for why a species is declining. It helped my career and helped the USFWS make better management decisions about the species. It also fostered new collaborations with international colleagues.”
“The lack of a definitive result changed our methodology and facilitated a collaboration with another group to further explore alternatives”
Strengthened their reputation as a trustworthy researcher who acts with integrity and transparency, or as a pioneer in their field
“The most prominent case where I published null results, I was presenting data that showed that an area of research in coal desulfurization was only being pursued because of false positive results. After this was published, research in that particular area essentially ceased. As a result, while this paper was widely read, it was not cited to any great extent because there were no papers being published to cite it. On the other hand, it did enhance my reputation as a careful, trustworthy researcher.”
Had a positive impact on career progression
“My published manuscript of 'null' results including discussion of potential reasons for null results and implications for future research, was selected for a national award as “best published manuscript" for the year (ADHD study) by the peer reviewed journal that published study results.”
“[…] It also helped me secure a PhD position.”
Built with Shorthand