Generative AI and the Future of Research Assessment — Insights from the REF-AI Report

by ChatGPT-5.1

The REF-AI report provides the first large-scale, qualitative national study of how universities across the UK are experimenting with, resisting, or preparing for the arrival of generative AI (GenAI) in the Research Excellence Framework (REF2029). Drawing upon more than 40 hours of focus groups, 30 hours of senior-leader interviews, a national survey, and a review of emergent research literature, the report paints a vivid portrait of a sector both cautious and compelled: cautious about risks to research integrity, but compelled by unprecedented administrative pressures, escalating costs, and rapid technological change.

Across 16 universities, the study reveals an uneven but unmistakable trajectory toward AI-augmented research assessment. While current adoption is “relatively shallow and patchy,” the gravitational pull of GenAI is unmistakable, and there is widespread recognition that REF2029 cannot ignore — and in many ways, has already been overtaken by — the datafied ecosystem in which UK research now operates. The REF-AI report, therefore, becomes more than a study of institutional perceptions; it becomes a diagnostic of a system at an inflection point, forced to reconcile long-standing traditions of peer review with the realities of algorithmic automation.

1. Key Findings

1.1 Patchy adoption, uneven readiness, and institutional inequality

The study finds that experimentation with GenAI varies dramatically, with resource-rich universities developing in-house AI tools or piloting REF-specific prototypes, while others have neither capacity nor appetite to engage. The result is a widening capability gap. Institutions with sophisticated digital infrastructures are already using GenAI for output selection, narrative drafting, or internal calibration — often seeing this as a way to gain a competitive edge. Conversely, smaller or less-resourced HEIs fear being locked out of a REF ecosystem that increasingly rewards AI maturity.

1.2 AI as burden-reducer — but not a silver bullet

Participants consistently agreed that GenAI may help alleviate the extreme labour costs and inefficiencies of REF processes, particularly in data management, summarisation, and internal quality checks. However, the report warns that AI should not be confused with a systemic cure-all. Instead, AI is becoming woven into the very architecture of research governance — “unavoidable, perhaps indivisible” — but still dependent on careful human oversight.

1.3 Automation of assessment is seen as inevitable

One of the most striking findings is the widespread belief that REF panels will use GenAI, whether formally permitted or not. Many respondents predicted a “partially or fully automated REF” in due course, driven by assessment burden, time constraints, and improved AI performance by 2029. Efforts to ban AI are widely expected to result in covert, unregulated use.

1.4 The role of AI in peer review: contested but accelerating

While universities are experimenting with GenAI to prepare outputs, impact case studies, and People–Culture–Environment statements, academics remain deeply concerned about erosion of scholarly judgement, the dilution of peer review, and machine-driven bias. Yet paradoxically, REF itself is perceived as already a compromised form of peer review — a “comfort blanket,” ritualised rather than rigorous, and deeply shaped by prediction of panel behaviour rather than direct scholarly appraisal.

1.5 A vacuum of governance and policy

Despite ongoing experimentation, the report finds “an existing dearth if not general absence” of institutional policy governing AI use for REF. Practice is running far ahead of governance. Few HEIs have defined permissible use, audit requirements, or disclosure obligations. Training is inconsistent. AI literacy is low. Many professional services staff are enthusiastic adopters; many academics remain sceptical or resistant.

1.6 Literature review confirms the ‘support tool, not substitute’ consensus

The accompanying literature review underscores convergence in the research community: LLMs can aid summarisation, screening, consistency checks, reviewer selection, and early-stage analysis, but cannot replace expert judgement. Challenges include hallucinations, disciplinary insensitivity, bias reproduction, opacity, and unresolved legal and ethical issues.

2. Most Surprising Findings

2.1 Near-universal expectation that panel members willuse AI

Despite public caution from funding bodies, respondents express near consensus that REF2029 panelists — many of the UK’s leading researchers — will inevitably rely on AI tools, formally or informally. Any assertion to the contrary is met with “scepticism and suspicion”.

2.2 AI adoption enthusiasm is highest among those whose jobs are most at risk

Professional services staff — whose roles may be significantly rationalised by AI — are more supportive of its adoption than academics. This overturns common assumptions that labour-threatened groups resist automation.

2.3 AI is viewed as a means to improve institutional memory

Beyond efficiency, institutions see AI as a strategic asset for codifying, storing, and retrieving institutional knowledge, especially around REF processes that recur only once every seven years.

2.4 The REF itself is perceived as outdated due to advances in AI

Participants argue that technological change has overtaken the REF’s conceptual footing. By comparison to emerging data-driven governance models internationally, the UK risks appearing antiquated if REF2029 does not modernise.

3. Most Controversial Findings

3.1 A “fully automated REF” may be desirable — not merely possible

Some interviewees suggested that automating REF assessment could free leading scholars from burdensome evaluation duties and allow them to focus on research itself. This challenges deeply held convictions about peer review as a human-centered process.

3.2 The REF’s claim to human judgement may become performative

If AI is widely used behind the scenes, the “human judgement” narrative underpinning REF legitimacy could become symbolic rather than substantive — raising transparency and trust issues.

3.3 Standardised national AI systems may reinforce inequality

While many advocate a national REF-AI tool to ensure equity, others fear a subscription-tiered system or a one-size-fits-all design that disadvantages niche disciplines or AI-mature institutions, creating a new digital divide.

3.4 Environmental cost of GenAI contradicts sustainability commitments

AI adoption for REF preparation may conflict with universities’ climate pledges, introducing a tension between institutional values and technological necessity.

4. Most Valuable Findings

4.1 Clear articulation of where AI genuinely adds value

The report identifies high-value, low-risk applications for GenAI:

• summarising long documents

• consistency checks and calibration

• data extraction and formatting

• support for drafting narratives (impact, PCE)

• identifying discrepancies or missing evidence

• supporting panellists in validation steps

These applications extend human capability without replacing scholarly judgement.

4.2 Explicit conditions for responsible integration

The recommendations provide a blueprint for national AI governance in research assessment, including:

• institutional AI policies

• secure environments

• disclosure statements

• quality assurance checklists

• audit logs

• equity provisions across HEIs

• compulsory AI literacy training for panelists and REF leads.

4.3 The recognition that REF automation is a systemicissue, not a technological one

The adoption of GenAI forces a reckoning with the REF’s overall design: its workload, evaluation philosophy, competitive incentives, and the culture of academic labour.

5. Recommendations

5.1 For AI Developers

1. Develop REF-compatible, secure, sector-specific LLMs
   Models trained on licensed, curated, domain-specific corpora can mitigate hallucinations and bias. Closed, auditable systems are essential.

2. Build explainability and provenance features by design
   Transparency is foundational for research assessment legitimacy.

3. Prioritise energy-efficient architectures
   Sustainability must accompany capability improvements.

4. Enable robust logging, version control, and reproducibility
   These are critical for audit trails in assessment contexts.

5.2 For Publishers

1. Provide legally compliant, high-quality training datasets
   The REF relies on accurate content analysis; publishers can supply structured, rights-cleared corpora to reduce legal and ethical risk.

2. Develop AI-ready metadata and machine-readable research objects
   Enriched metadata will improve accuracy in automated evaluation contexts.

3. Collaborate on sector-wide standards for citation, provenance, and transparency
   Publishers play a central role in establishing norms for how AI may handle scholarly texts.

4. Prepare for a future in which REF evaluators may use AI to interrogate outputs
   Designing content formats that are machine-navigable and human-interpretable will be essential.

5.3 For the Scientific Community

1. Adopt a blended model of AI-assisted but human-verified assessment
   AI should support, not replace, scholarly judgement.

2. Invest in AI literacy across all career stages
   Understanding AI is now a foundational research skill.

3. Maintain peer review integrity while modernising workflows
   The community must define what scholarly judgement means in an AI-rich world.

4. Address inequality head-on
   Institutions with fewer resources must not be structurally disadvantaged by the shift toward data-driven assessment practices.

5. Engage in open debate about the philosophical purpose of the REF
   Before AI reshapes the REF by force, the academic community must articulate what it wants the REF to be — and what values it must protect.

Conclusion

The REF-AI report offers a rare moment of clarity in a rapidly shifting landscape. It exposes a sector pulled in multiple directions: anxious about technological disruption, sceptical of AI’s limits, yet acutely aware that REF2029 cannot be insulated from the rising tide of automation. The report’s central message is neither alarmist nor techno-optimistic. Instead, it calls for governed innovation: principled experimentation, transparent reporting, equitable access, and unwavering commitment to human expertise.

REF2029 will not be the last research assessment to grapple with AI. But it may be the first to define — through governance, not accident — how AI and academic judgement coexist. Whether the REF becomes a stronger, fairer, more efficient system will depend not on AI alone, but on the policies, norms, and values the sector chooses today.

The research is clear: the future is not “AI everywhere.” The future is “AI integrated through evidence, rights, and human expertise.”

PASCAL HETZSCHOLDT

·

27 NOV

AI, Public Infrastructure, and the New Social Contract: Synthesizing Copyright Governance, Health-Economics, and Learning Sciences in the Era of the Genesis Initiative

Read full story