News

Report published 12 December 2022.

Future Research Assessment Programme launch event: Machine learning, metrics & merit: the future of research assessment: 12 December 2022. Watch a recording of the launch event.

Can Journal Article Quality be Assessed by AI?

Overview

The Responsible Use Of Technology-assisted Research Assessment project is part of the The Future Research Assessment Programme, which "aims to explore possible approaches to the assessment of UK higher education research performance". It is led by the by all four UK higher education funding bodies. This page will link to all relevant documents, when published.

Main report on the potential uses of AI in future Research Excellence Frameworks (REFs) in the UK

This investigates whether there is a role for AI to support future REFs. ***Download the main report here: Can REF output quality scores be assigned by AI? Experimental evidence. ***

Talk summarising the study.

List of recommendations from the main report

Literature review on AI in research assessment

Literature review: Reviews research related to possible AI automation of various REF tasks. Makes a list of separate recommendations for the future REF in terms of tasks that could be partly automated.

List of recommendations from the literature review.

  1. Implement a system to recommend sub-panel members to review outputs. This would likely be based on the ORCIDs of sub-panel members matching their Scopus/Web of Science/Dimensions/etc. profiles, then using text mining to assess the similarity of their outputs with each sub-panel output to be assessed. The text mining might use article titles, abstracts, field classifications and references.
  2. Build for the long-term implementation of quality control systems for academic articles by recommending that preprints of outputs for the next REF are saved in format suitable for text mining. Ideally, this would be in a markup format, such as XML, rather than PDF. This will also help longer-term AI systems for predicting REF journal article scores with article full text processing. At the end of the next REF, a future technology programme could then investigate the potential for full text mining for quality control purposes (e.g., checking statistics, plagiarism checks).
  3. Build for the long-term exploitation of open peer review by, at the end of the next REF, calling for a review of current progress in the use of AI to exploit open peer review to assess article quality. Whilst open peer review should not be used as an input because it can be too easily exploited, investigations into its properties might shed useful light on aspects of quality identified by reviewers. Research into this is likely to occur over the next few years, and a review of it near the next REF might provide useful insights for both future AI and future human peer review guidelines for sub-panel members.
  4. In the next REF, collate information on inter-reviewer agreement rates within sub-panels for outputs scored before cross-checking between reviewers. Use this to assess the human level agreement rates (for all output types) to use as a benchmark for score prediction AI systems.
  5. In the tender for bibliometrics and AI for the next REF (if used), mention the importance of accurate classification for bibliometric indicators, including for the percentile system currently used.
  6. Warn sub-panel members of the potential for small amounts of bias in the bibliometric data and AI (if used) and continue with the anti-bias warnings/training employed in REF2021.

Additional reports

A series of additional reports give findings related to research quality of journal articles, covering related aspects such as bibliometrics, journal impact factors, collaboration, funding, altmetrics.

Main additional outputs on AI automation in the REF

Additional analyses investigating aspects of REF scoring

Additional analyses for wider UKRI policy

In the press

Nature: AI system not yet ready to help peer reviewers assess research quality

UKRI: Evaluation reports steer away from ‘automated’ UK research assessment

THE: REF robot reviewers ‘not yet ready’ to replace human judgement

ResearchProfessional News: AI not so smart at speeding up REF, experts conclude

THE: Funders mull robot reviewers for Research Excellence Framework

Nature: Should AI have a role in assessing research quality?