Opinion
In 2024, Larry Richardson had built a Google Scholar h-index of 11 over a short stretch of time, highlighting his promising trajectory as an up-and-coming scholar. The only problem was that Larry turned out to actually be a cat, and his Google Scholar profile was created by his human owner – Reese Richardson – to highlight how easy it is to manipulate Google Scholar citation metrics.
Problems with the validity of citation-based data are not restricted to Google Scholar but extend to curated and authoritative databases such as Web of Science. Each year, a growing number of ‘highly cited scholars’ are excluded from Clarivate’s annual list of Highly Cited Researchers due to anomalies resulting from, inter alia, hyper-prolific authorship and other strategic behaviours aimed at gaming citation counts. In 2025, the number of researchers thus excluded had reached a number of about 2,400 – a rather high number compared to the 6,900 scholars ultimately deemed legitimately ‘highly cited’ by Clarivate.
Both of these examples illustrate the considerable problems associated with using citations as an indicator or measure of research performance. In many ways, this should not be surprising. The more, over recent decades, citations have attained a central role in assessments of research and in the various rankings produced on their basis, the more these also have come under pressure from a range of strategic and manipulative behaviours. In many ways, citations thus are a prime example of Goodhart’s Law in action – once citations became a target, they ceased to be a good measure.
Well-documented activities aimed at inflating citation counts range from downright fraud – the purchasing of citations or metadata citation manipulation being key examples – to more inconspicuous forms like excessive and strategic forms of co-authorship.
Even 15 years ago, scientists at Australian universities we interviewed for a research project likened co-authorship to a ‘game’ one gets into to boost publication and by extension, citation counts, without doing necessarily more work. In the meantime, this behaviour has spread to the humanities and social sciences too. This is also since, including in the Australian context, universities’ performance management frameworks tend to encourage co-authorship particularly if involving external collaborations. The key reason for this is the citations flowing from such collaborations can boost a universities’ rankings performance in ways that have rightly been deemed absurd. It does not help that the major university rankings have so far generally failed to properly adjust for this through fractionalizing citation counts, despite this having been long called for.
In addition to citations being distorted by myriad strategic activities, there is also another reason why citations may be well past their use-by date as a measure of research performance. To considerable extent, the legitimacy of citations as a measure is contingent upon the cited source having been properly engaged with and carefully evaluated by the citing authors. That this is generally the case has long been doubted. In 1965, Norman Kaplan in a classic paper questioned the fundamental idea that citations generally reflect “a careful and accurate” recognition of previous working informing the research presented. More recently, a wide range of studies have provided evidence that a considerable proportion of citing papers engage with the papers they cite either not at all or if then only rather superficially. This phenomenon appears to equally apply to disciplines that pride themselves of their careful use of sources, such as history.
It is obvious that all of this will only get worse with proliferation of AI-produced text in the published literature. It is not unfeasible that already in the near future, the majority of citations appearing in this literature will not have involved any in-depth human scrutiny of the citing source. Adding to this, it also has been suggested that when it comes to citations, the use of AI may further reinforce a form of popularity-based Matthew effect, through adding to the “tendency for highly cited work to accumulate even more citations while lesser-known research remains invisible”.
Reflecting on all of these developments, it is hard to see how citations really can have a long-term future as a measure or indicator of research performance. Maybe, for a little bit longer, their use will be stabilized by their inclusion in university rankings and by the uncritical obsessions of parts of the academic community. But, beyond that, the writing is on the wall that citations as a measure are moribund. One can only hope that universities including those here in Australia have the courage to recognise this in time, and move beyond a simplistic focus on citations or associated measures when internally assessing or incentivizing research. Otherwise, governments – reflecting their own concern with research impact – may ultimately force them to.
Associate Professor Peter Woelert works in the Faculty of Education at the University of Melbourne