Researchers’ views on assessment practices

In this part of the survey, we asked respondents to think about the types of evaluations they had recently experienced, as outlined earlier in Figure 1. Overall, perspectives were positive (Fig. 9), and this trend was consistent amongst all regions (though at differing levels, ranging from 56-73% net-positive in Europe to 82-88% net-positive in Africa). 

Figure 9: Whether or not it was successful, did you feel positively about your experiences of the following types of evaluation? (n=5,291) 

*Numbers may not add up to 100% due to rounding

This is perhaps surprising considering Part 1 of this white paper, where we found that many activities researchers commonly undertake are not considered within evaluations. When we look at views on assessment practices more broadly, a more nuanced picture emerges. On the whole results remained positive, with only 20% of researchers indicating dissatisfaction with the way their research is evaluated. However, a significant proportion of respondents expressed neutral views (Fig. 10), and in Europe and Oceania negative views were more prevalent. 

Figure 10: To what extent do you agree with the following statements? (n=5,201) 

*Numbers may not add up to 100% due to rounding

Within free-text comments (n=1,279), some researchers had substantive criticisms of the assessments they experience. These can be summarised as follows: 

Concerns relating to trust in the process, including a lack of transparency, inadequate experience of evaluators, and bias and subjectivity within evaluation practices.

“The measures used for research evaluation are usually not objective and they can be manipulated so that faculty members can choose their preferred candidates for promotion, hiring etc.”

Switzerland, early career stage, multiple disciplines

An overreliance on publication metrics leading to a focus on quantity over quality, as well as a disadvantage to certain disciplines, niche or interdisciplinary research.

“It is suggested to comprehensively consider the various abilities and qualities of researchers, rather than relying solely on published papers and funding ... Sometimes, overcoming a difficult problem may take a long time during which there may not be any research output.”

China, postgraduate student, Biological & Life Sciences

Frustration that research assessment often judges things that a researcher cannot control or influence – relying on the downstream impacts of their work compared to the quality and effort they put in.

“The evaluation criteria are based on quantitative indicators or on indicators that are not depending on my effort and that I cannot improve by simply working more and better.”

Romania, established career stage, Physical Sciences & Engineering

The difficulty in assessing the societal impact of research due to inadequate measures or indicators with broad applicability.

“I think most evaluations miss the genuine impact of research. However, I am aware that that is a very difficult thing to do systematically.”

South Africa, established career stage, Biomedical Sciences

Many of these are in line with our findings in Part 1 of this white paper. However, we were interested to find that a significant number of researchers expressed concerns about qualitative forms of evaluation which, as discussed, are considered by research assessment reform initiatives to be fairer indicators. This could also go some way to explaining the positive views on existing (quantitative) approaches, though more research would need to be done to validate this theory. 

As we have already discussed, there are several initiatives within the research community that advocate for more transparent and inclusive research assessment methods. To understand how these movements have affected institutional practices, we asked researchers to identify whether their institution was signed up to any initiatives, and whether they had observed any changes in research assessment as a result.

Awareness was low, with around half of researchers (n=5,039) stating that they do not know whether their institution is signed up to any initiatives; this ratio did not change when segmenting the data by region or career stage.  

For those who stated that their institution is signed up to one of these initiatives, we then asked whether they had noticed any changes in the way research assessment is carried out. Responses were mixed, with 19% of researchers perceiving significant or very significant change, 27% perceiving some change, and 40% perceiving little to no change (Fig. 11). 

Figure 11: Do you perceive any changes in the way that research assessment has been carried out in your institution since signing up to any initiatives? (n=1,290) 

*Numbers may not add up to 100% due to rounding

Within free-text comments (n=366), respondents stated that their institutions acknowledged the need for change, and tended to communicate this during evaluation processes. However they also recognised the difficulty in implementing these changes at all levels – especially when it comes to disincentivising ingrained behaviour on a large scale. 

“The DORA principle is invoked whenever we interview people, which is good. However, it is hard to eradicate the instant pull of a Nature, Science or Cell paper on a CV for senior researchers of my generation.”

UK, established career stage, Biological & Life Sciences

“The importance of benefit to society is acknowledged as is the need to change the way we are evaluating. The will is there, but the methods are lacking.”

Netherlands, established career stage, multiple disciplines

“We are instructed to keep a holistic view of research evaluation and not consider metrics like h-index. However, most evaluators still keep that in mind, and let it guide qualitative assessments.”

United States, established career stage, Natural Sciences

When we asked researchers about their opinions on the workload involved in research assessment practices, along with access to institutional support, results were quite varied – with around 45% positive responses compared with around 35% negative (Fig. 12). As with other questions, researchers in Europe were the most pessimistic, with negative views rising to around 45%. 

Figure 12: To what extent do you agree with the following statements? (n=4,284) 

*Numbers may not add up to 100% due to rounding

In the free-text comments (n=922), researchers had several suggestions on how their institutions could help them prepare for evaluations, from which four common themes emerged: 

• Simplification and standardisation of processes to make it easier for researchers to understand requirements and reduce duplicated work. 
• Increased use of systems and technology to streamline or automate manual tasks. 
• Improved administrative support for evaluation activities. 
• A reduction in workload and institutional bureaucracy in general, to give researchers more time for both conducting research activities and evaluation of research. 

“We need to feed data to different automated systems (databases) that do not connect to each other. Interconnection among these systems would greatly improve how much time is spent on evaluation reports.”

Brazil, established career stage, Biological & Life Sciences

“Simplifying reporting requirements and providing clear guidelines of what constitutes acceptable evidence of impact would streamline the process.”

India, postgraduate student, Physical Sciences