I’ve recently taken part in a couple of discussions about the kinds of papers and books that are highly ranked in quality reviews, reviews like the UK’s REF. Those in the discussions aimed to understand what counts as “outstanding”. This aim wasn’t simply borne from an instrumental interest, getting clear about what to do in order to do better. Some of us were equally interested in sharing more critical perspectives.

You see, in the UK where I’m based, most people feel they have no choice but to be involved in these kinds of audit-oriented discussions. This is largely because there is such a lot riding on the outcomes. Government funding for research – and for doctoral education, student enrolments and reputation – all now appear to be permanently entangled in audit exercises. Understanding this ac-counting game- to use the Bourdieusian metaphor of a game – getting into it, playing in it and with it, perhaps even being able to change it  – is now an inextricable part of UK academics’ work. Many of us do end up on a tightrope of with/against politics in order to do our bit for our institution, at the same time as at least trying to voice our concerns about the need to make the sector fairer for everyone.

So in the interests of understanding the game, which does not signal my approval of it you understand, here are a few thoughts about what in the UK is now known as the 4* publication, those deemed by ‘peer reviewers’ to be the best, to be “outstanding”. The ideas I report here are not just my own, but come from those conversations I mentioned, the ones held in the last few weeks. These comments are also largely social science in orientation, even though my own work straddles the arts and much of it doesn’t get a look in in disciplinary-based audit exercises. That’s another story and another post! So let me get on with “outstanding”…

As a general rule, the “outstanding” publication usually isn’t “outstanding” just because it’s in a highly ranked journal – there are a couple of disciplinary exceptions to this rule, but by and large this is the case in social science. So if the quality judgment is not about the journal it must be about the actual characteristics of the paper or book itself. Well that’s certainly what we are told.

So here’s the rub. Usually, an “outstanding” paper:

• pushes the field forward – it is work at big scale and/or it has big data and/or big ideas and/or deals with big problems. It is ambitious.
• is explicit about the significance and the nature of the contribution (empirical, theoretical, challenging long held assumptions etc.)
• is situated within a deep understanding of international literatures, and the histories and debates within the field
• is written in a highly authoritative voice, and it is well written
• may or may not challenge traditions and genres
• provides convincing evidence and/or argument
• is potentially of wide interest, could be set as recommended or essential reading in course work
• may well have had the benefit of very experienced reviewing.

Nothing to it then!! The issue for most of us is of course how to find the time and also, let’s face it, the chutzpah to try to fulfill this kind of brief. But as we often say about the PhD, it’s not a Nobel prize and “outstanding” is actually a normative Bell curve kind of judgment… so it is within the grasp of a (select) number of ‘jobbing academics’.

Saying this of course doesn’t mean that we don’t all feel trapped in some kind of endless performative exercise, like this mouse on a spoon. It’s more the case that the REF game is one that some of us can just about manage and tolerate, as long as the cat goes off for a bit of a nap now and then.

Conversations about “outstanding” are inevitably accompanied by discussions of the centralised process of audit. And these are highly political in nature. For instance:

• designated peer ‘scholar auditors’ are always a selection from the disciplinary community. They may be more or less conservative in their views. As ‘quality gatekeepers’, they may value some publications, publication outlets and types of research more than others and this will play out in their awarding of ‘ratings’ to the work that they see and read. Even within the same discipline, one panel’s “outstanding” may not be another’s.
• universities operate very and various selective processes, choosing which work to submit for audit and review. Their decisions are always about managing the tensions between their income and reputation. Many institutions also use audit selection exercises as part of their performance management processes, even if informally.
• the nature of the audit exercise itself is always a process enmeshed in wider government games – managing national reputation, balancing credibility with efficiency, being seen to operate a credible “hands off” exercise at the same time as operating a system designed to (re)produce hierarchies of institutions.

You’ll notice that I’ve scare-quoted “outstanding” all the way through this post because I want to signal I know the problematic nature of the term. What counts as “outstanding” at any one time is not a fixed and arbitrary “standard”, regardless of how it is judged and measured and by whom, and producing an “outstanding” publication list is not all that there is to academic work.

It’s crucial to note that scholarly work that isn’t “outstanding”- according to these criteria – can still be really important. Healthy disciplines produce a variety of scholarly work, for various purposes and audiences. A small-scale evaluation of a local service mightn’t rate as “outstanding” in the quality audit, and it may not change government policy or be set reading in every first year course in the country (that is, it mightn’t have “impact”), but it might still change the lives of local people. That’s worth doing. A niche project looking at a particular series of films made a long time ago may not push the field forward, but it might be a crucial building block to a growing body of knowledge which eventually becomes significant. That’s also worthy of scholarly time.

In my view it would be a pretty disappointing social science school which skewed its entire effort towards the production of “outstanding” papers. Such a school would be doing a real disservice not only to the discipline, but also to the purposes of scholarship – not to mention the difficulties it would surely create for ‘career progression’ (as difficult as this now is).

Advice to early career scholars, and those trying to make their way through the competition for academic work, often fails to recognise the highly political nature of the ways in which judgments about quality of publications are made. However, it is still useful for early career researchers to consider, within their discipline(s), the more general characteristics of “outstanding” texts and to think about how they might aim to produce some of this kind of work. And they might reasonably expect, if they are employed, to get support from their schools/faculties in this venture; precariously employed scholars inevitably have to look to learned societies as well the sporadic and scattered advice online. I do wonder how much this is the case.