Opinion
It is no secret that the academic publishing system, and the journal system as its core pillar, is in severe crisis due to the onslaught of AI-generated work.
However, it should not be forgotten that even before the advent of AI, the cracks were already clearly visible. Key symptoms include the explosion in both journals and article numbers, often driven by financial or strategic rather than scholarly considerations, false or manipulated citations, the proliferation of paper mills and predatory journals undermining trust in the published research, and escalating costs for universities and ultimately the taxpayer partially due to the shift to open-access publishing.
AI has added significant fuel to the fire. Everyone who knows someone who is editing an academic journal has heard stories about the flood of AI-generated article submissions academic journals receive these days. In one telling instance I was told about recently, an author had submitted eight articles to one and the same journal in a short stretch of time. All of these articles looked AI-generated and were well-written at face value but upon further scrutiny revealed to make little in terms of a substantial contribution. It is increasingly clear that the journal system is incapable of processing this volume of AI-generated submissions while also maintaining some semblance of effective quality assurance.
This is also because, peer review – one of the contested yet long-established pillars of the quality assurance system of the modern journal system – is in severe crisis. As peer-review remains little incentivised by universities’ ‘publish or perish’ approach or publishers’ lack of compensation, academics appear to be increasingly reluctant to act as peer-reviewers. Others resort to using AI to generate their reviews. Tellingly, on the day of writing this piece I came across a social media post of someone complaining that they had just received two reviewer reports from a journal that were nearly identical down to the wording, both evidently AI-generated. The crisis of peer review thus is at the heart of concerns about the sustainability and standard of quality assurance practices in the journal system, now and into the future.
While the role of AI as a catalyst in the crisis of the journal system has recently received ample attention, the same is less so the case when it comes to the likely implications for university rankings.
Just as with the journal system, university rankings have long had their critics. Such criticism extends to the ostensibly more ‘objective’, metric-based forms of research assessment that are a core pillar of most university rankings. This includes, for example, long-established specific methodological critiques, such as those aimed at rankings’ general lack of use of fractional counting of publications and citations when it comes to co-authored publications involving authors from different institutions.
More generally, however, one of the most striking vulnerabilities of the metrics-based elements of university rankings remains their dependence on the academic publishing system more broadly and on the journal system more specifically. To varying degrees, all major university rankings derive their key ‘objective’ research metrics such as publication volume or number of citations from databases that have long been dominated by the journal system. All of this makes, by implication, the crisis of the journal system a crisis afflicting university rankings also. A journal system whose quality assurance mechanism appears to be collapsing, and which is further increasingly plagued by citation noise, evidently does no longer provide a trustworthy source on which to base any reliable comparative assessments of research performance on.
What does this mean for the future of university rankings? At the present, the perhaps most likely scenario is that the can will be kicked down the road a little longer. This is while the journal system will be increasingly populated with AI-generated manuscripts that have been reviewed by AI also. For the time being, universities will continue to strive for increases in publication output and number of citations and will celebrate their ranking performance in their usual selective manner.
However, given the current state of crisis, it also looks increasingly likely to me that we will eventually see the ground shifting in the not-so-long-future because the AI-associated pressures and tensions manifesting within the journal system will become too obvious to ignore.
Adding to this, universities are increasingly asked to demonstrate their societal benefit when it comes to their research. Within such context, it becomes increasingly hard to argue that what the world really needs is more academic papers. Last not least, given the impact of AI, it is not unconceivable that at some stage in the future, universities’ real value proposition will be the critical evaluation and testing rather than the generation and publication of knowledge.
Reflecting all of this, we may not be far away from a stage where university rankings in their current form relying upon traditional research metrics will cease to exist. In anticipation of this, one can perhaps expect to see a sustained pivot to broader research impact indicators and the like in major rankings to signal a ranking’s continued relevance. Whatever this pivot may look like specifically, one thing remains certain: any new indicators will naturally become the focus of universities’ strategising that will undermine what these indicators purport to measure.
Associate Professor Peter Woelert is a researcher at the University of Melbourne.