Research Excellence Framework significantly increases UK university output but not efficiency, study shows

A new study led by City, University of London suggests that the introduction of the Research Excellence Framework (REF) to measure the research performance of UK universities has led to significant increases in the quantity of their high-quality published research, but not in the output of active academics, and that it must do more to support university research strategies.

Many countries around the world use performance-based research systems (PRFS) as a means of determining the distribution of funding for research across higher education institutions. In 2008, the REF replaced Research Assessment Exercises (RAE) as the measurement of this across UK universities, with the first results in 2014 to cover the period between 2009 and 2014.

The research by Giulia Iori and Albert Banal-Estañol, Professors in Economics at City, alongside academics from Universitat de Barcelona, London School of Economics and Political Science and University of Palermo, investigated the performance of institutions and the incremental effects of REF2014 on output and quality of the research submitted across UK universities in contrast with comparable institutions in the United States, which are not exposed to such a measure.

The sample included 103 UK universities that submitted research to REF2014’s ‘Economics and Econometrics’ panel or ‘Business and Management’ panel. As a control group, 135 US universities with a top-quartile Department of Economics or Business School according to the Research Papers in Economics (RePEc) database from December 2018.

Key findings from the research include:

• REF2014 significantly increased UK universities’ research output compared to US institutions, but the number of publications per author did not grow with it. This suggests the REF generally encouraged universities to hire more academics to increase activity rather than supporting their existing researchers to produce a higher volume of work.
• Subsequently, the number of UK university publications increased on average per department by 41.37 per cent across the time period, with the most pronounced change taking place towards the end of the cycle, between 2012 and 2014.
• Research excellence – measured by publications in top journals (3*, 4* and 4** journals as measured by Association of Business Schools’ classification of scientific journals) – also increased relative to other countries following implementation of REF.
• Increases in research output for REF2014 were generally more pronounced for Russell Group universities than non-Russell Group ones, with a stronger positive impact from REF2014 in terms of number of top publications per author.
• Submissions from universities that selected the Economics and Econometrics RAE2008 panel but switched to the Business and Management panel for REF2014 were significantly decreased compared to those that remained with the same panel. Consequently, their proportion of business publications in top journals did not increase.

Researchers used Scopus, the largest worldwide database of peer-reviewed literature, to obtain all articles that included at least one of the 103 UK universities or 135 US universities as an affiliation, had either ‘Economics, Econometrics and Finance’ or ‘Business Management and Accounting’ as subject areas, and were published between 2001 and 2014. Institutions submitting fewer than ten papers per year on average between 2001 and 2008 were discarded from the overall sample size, which totalled 145,536 unique publications.

Professor Iori said the study suggested REF needed to do more to encourage universities to invest in processes and supportive strategies.

“The Research Excellence Framework is an important tool for universities to be able to attract funding, recruit students and increase their national and global prestige,” she said.

“When compared to US universities, our data shows that the REF incites a higher number of research publications and articles published in the best journals. However, given the number of publications per author remained stable throughout the research period, this appears to be driven by institutions hiring more academics rather than an intrinsic motivator for academics themselves to publish in leading journals. This recruitment is simply a redistribution of academics rather than an enhancement of research capacity.

“Ultimately this benefits the elite universities who can attract the best academics, and hurts the lesser ones who are losing them to competitors, but does little to improve overall standards.”

Professor Banal-Estañol agreed that the study highlighted areas of improvement needed in the assessment framework.

“The REF must encourage greater collaboration between domestic competitors, rather than pitting them against each other,” he said.

“Comparing universities domestically is not highly meaningful as the best ones will always outperform the worst, especially with the polarisation of academic redistribution. If too much of a gap exists, this can discourage weaker institutions from producing and submitting research which has an overall detrimental effect.

“Instead, the REF needs to reward universities that outperform others on a similar trajectory, rather than pitting strong against weak, which is why international comparisons are much more useful for driving overall standards.

“This study provides the basis for alternative measures of success of a performance-based research system, to ensure the fairest means of distributing funds while preserving competition and increasing overall standards.”

In City’s latest submission to the REF (REF2021), 86 per cent of research was rated as being of world-leading (4*) and internationally excellent (3*) quality.

‘Performance-based research funding: Evidence from the largest natural experiment worldwide’ by Professor Albert Banal-Estañol, Professor Giulia Iori, Professor Mireia Jofre-Bonet, Laia Maynou, Michele Tumminello and Pietro Vassallo is published in Research Policy.