The future of research assessment

In the concluding section of the survey, we wanted to look towards the future by exploring researchers' preferences for research assessment. 

Using the categories defined in the introduction of this white paper, we asked respondents to judge how these categories are currently taken into account during assessment processes, and compare this with their ideal scenario.  

When we aggregated these results, research outputs remained prominent even in an ideal scenario (Fig. 13). However, overall researchers expressed that their ideal evaluation is one that moves towards a greater weighting on their contributions to society (referred to in the survey as “the public good”), and wider contributions to research and research culture – this trend was consistent across all regions, disciplines, career stages, and types of research.  

Figure 13: How would you say the importance of each of these areas is weighted in terms of how you are currently assessed, and in an ideal evaluation process what would you suggest should be the weighting? (n=6,616)

*Numbers may not add up to 100% due to rounding.

“A comprehensive and balanced evaluation process is crucial for fostering high-quality research and recognising the diverse contributions of researchers.”

Algeria, mid-career stage, Physical Sciences & Engineering

A similar shift was observed when we asked researchers about their preferences for the usage of quantitative and qualitative indicators during assessment processes, with 52% of respondents indicating a preference for an equal balance between the two methods (Fig. 14). When compared with Figure 2, where most researchers identified metrics as the dominant feature of their current assessments, we can see a clear desire to reduce reliance on these metrics. This is the case regardless of region, career stage and discipline. 

Figure 14: In an ideal research assessment process, what would you suggest should be the weighting of quantitative and qualitative methods? (n=4,869) 

It’s worth noting, though, that a sizeable proportion of researchers still identified a metrics-based approach as preferable: 27% stated that their ideal evaluation was entirely or mostly based on metrics, compared to 18% preferring entirely or mostly qualitative evaluations. This was similar across all demographics, and is likely related to concerns about perceived bias and increased workload involved within qualitative assessments, as we found in previous questions. 

This again highlights a disparity between researchers’ views and initiatives for research assessment reform, but also acts somewhat in conflict with researchers’ preferences within the previous question. If we look at the survey data, we see that researchers would like evaluation processes to better consider their contributions outside of the research outputs they publish – but would still like metrics to form a significant part of these processes. This leads to a contradiction, because these types of contributions are much harder to measure and evidence through metrics, and any moves to include these categories more within assessment processes will naturally lean towards the qualitative. 

“Ideally, research evaluation should strive for an approximately equal balance between quantitative and qualitative methods. Metrics provide a starting point, while qualitative descriptions enrich the narrative and offer a more holistic understanding of the research’s value.”

Iran, established-career stage, Clinical Medicine & Healthcare 

Finally, we asked researchers to consider which elements of their work would form part of their ideal evaluation process – again categorising these into research outputs (Fig. 15), public good activities/positive contributions to society (Fig. 16) and wider contributions to research (Fig. 17) and the research community (Fig. 18). There were 16 elements cited by over 50% of researchers, spanning all categories, which further emphasises the desire for a balanced approach to research assessment. 

Figure 15: Which aspects of your research outputs do you feel should be considered in an ideal evaluation of your contribution? (n=4,945) 

Figure 16: Which aspects of “public good”* do you feel should be considered in an ideal evaluation of your contribution? (n=4,907) 

* “Public good” was defined throughout the survey as “how your research contributes to and influences society, the economy, and the environment beyond academia” 

Figure 17: Which aspects of your wider contribution to research do you feel should be considered in an ideal evaluation of your contribution? (N=4,756) 

Figure 18: Which aspects of your wider contribution to the research community do you feel should be considered in an ideal evaluation of your contribution? (n=4,756) 

When compared to current assessment practices in the current status section of this white paper (Figures 4-7), a few notable observations emerge: 

• Publication metrics remain desirable in assessing research outputs, with 74% of researchers stating that this would be included in their ideal evaluation. Interestingly, 64% also stated that the Impact Factor of the journals they publish in should be considered, despite this being widely challenged by initiatives for research assessment reform.   
• However, researchers’ ideal evaluation of their outputs more often includes other criteria – the most notable jumps being open access/interdisciplinary outputs and alternative metrics, raising from 14-20% to 30-37%. 
• In line with other sections of this white paper, researchers consistently want their contributions to the public good included within assessments, with each activity appearing significantly more in an ideal evaluation versus what is currently happening. Only 5% of researchers believe that these activities shouldn’t be included in assessment at all. 
• Similarly, most researchers would like their wider contributions to research considered more during assessments. Grant-related indicators (applications made and income received) were notable as the only two activities that were less prominent in researchers' ideal evaluation (compared to existing practices).