I’ve recently taken part in a couple of discussions about the kinds of papers and books that are highly ranked in quality reviews, reviews like the UK’s REF. Those in the discussions aimed to understand what counts as “outstanding”. This aim wasn’t simply borne from an instrumental interest, getting clear about what to do in order to do better. Some of us were equally interested in sharing more critical perspectives.
You see, in the UK where I’m based, most people feel they have no choice but to be involved in these kinds of audit-oriented discussions. This is largely because there is such a lot riding on the outcomes. Government funding for research – and for doctoral education, student enrolments and reputation – all now appear to be permanently entangled in audit exercises. Understanding this ac-counting game- to use the Bourdieusian metaphor of a game – getting into it, playing in it and with it, perhaps even being able to change it – is now an inextricable part of UK academics’ work. Many of us do end up on a tightrope of with/against politics in order to do our bit for our institution, at the same time as at least trying to voice our concerns about the need to make the sector fairer for everyone.
So in the interests of understanding the game, which does not signal my approval of it you understand, here are a few thoughts about what in the UK is now known as the 4* publication, those deemed by ‘peer reviewers’ to be the best, to be “outstanding”. The ideas I report here are not just my own, but come from those conversations I mentioned, the ones held in the last few weeks. These comments are also largely social science in orientation, even though my own work straddles the arts and much of it doesn’t get a look in in disciplinary-based audit exercises. That’s another story and another post! So let me get on with “outstanding”…
As a general rule, the “outstanding” publication usually isn’t “outstanding” just because it’s in a highly ranked journal – there are a couple of disciplinary exceptions to this rule, but by and large this is the case in social science. So if the quality judgment is not about the journal it must be about the actual characteristics of the paper or book itself. Well that’s certainly what we are told.
So here’s the rub. Usually, an “outstanding” paper:
- pushes the field forward – it is work at big scale and/or it has big data and/or big ideas and/or deals with big problems. It is ambitious.
- is explicit about the significance and the nature of the contribution (empirical, theoretical, challenging long held assumptions etc.)
- is situated within a deep understanding of international literatures, and the histories and debates within the field
- is written in a highly authoritative voice, and it is well written
- may or may not challenge traditions and genres
- provides convincing evidence and/or argument
- is potentially of wide interest, could be set as recommended or essential reading in course work
- may well have had the benefit of very experienced reviewing.
Nothing to it then!! The issue for most of us is of course how to find the time and also, let’s face it, the chutzpah to try to fulfill this kind of brief. But as we often say about the PhD, it’s not a Nobel prize and “outstanding” is actually a normative Bell curve kind of judgment… so it is within the grasp of a (select) number of ‘jobbing academics’.
Saying this of course doesn’t mean that we don’t all feel trapped in some kind of endless performative exercise, like this mouse on a spoon. It’s more the case that the REF game is one that some of us can just about manage and tolerate, as long as the cat goes off for a bit of a nap now and then.
Conversations about “outstanding” are inevitably accompanied by discussions of the centralised process of audit. And these are highly political in nature. For instance:
- designated peer ‘scholar auditors’ are always a selection from the disciplinary community. They may be more or less conservative in their views. As ‘quality gatekeepers’, they may value some publications, publication outlets and types of research more than others and this will play out in their awarding of ‘ratings’ to the work that they see and read. Even within the same discipline, one panel’s “outstanding” may not be another’s.
- universities operate very and various selective processes, choosing which work to submit for audit and review. Their decisions are always about managing the tensions between their income and reputation. Many institutions also use audit selection exercises as part of their performance management processes, even if informally.
- the nature of the audit exercise itself is always a process enmeshed in wider government games – managing national reputation, balancing credibility with efficiency, being seen to operate a credible “hands off” exercise at the same time as operating a system designed to (re)produce hierarchies of institutions.
You’ll notice that I’ve scare-quoted “outstanding” all the way through this post because I want to signal I know the problematic nature of the term. What counts as “outstanding” at any one time is not a fixed and arbitrary “standard”, regardless of how it is judged and measured and by whom, and producing an “outstanding” publication list is not all that there is to academic work.
It’s crucial to note that scholarly work that isn’t “outstanding”- according to these criteria – can still be really important. Healthy disciplines produce a variety of scholarly work, for various purposes and audiences. A small-scale evaluation of a local service mightn’t rate as “outstanding” in the quality audit, and it may not change government policy or be set reading in every first year course in the country (that is, it mightn’t have “impact”), but it might still change the lives of local people. That’s worth doing. A niche project looking at a particular series of films made a long time ago may not push the field forward, but it might be a crucial building block to a growing body of knowledge which eventually becomes significant. That’s also worthy of scholarly time.
In my view it would be a pretty disappointing social science school which skewed its entire effort towards the production of “outstanding” papers. Such a school would be doing a real disservice not only to the discipline, but also to the purposes of scholarship – not to mention the difficulties it would surely create for ‘career progression’ (as difficult as this now is).
Advice to early career scholars, and those trying to make their way through the competition for academic work, often fails to recognise the highly political nature of the ways in which judgments about quality of publications are made. However, it is still useful for early career researchers to consider, within their discipline(s), the more general characteristics of “outstanding” texts and to think about how they might aim to produce some of this kind of work. And they might reasonably expect, if they are employed, to get support from their schools/faculties in this venture; precariously employed scholars inevitably have to look to learned societies as well the sporadic and scattered advice online. I do wonder how much this is the case.
I’ve a couple of videos and a (very good in my opinion) top tips about writing papers that we captured 18 months ago at a Researchers’ Forum http://thedigitaldoctorate.com/2014/01/27/how-to-write-an-internationally-excellent-paper/
What’s really interesting here is the contrast in attitude between the natural scientist and the social scientist. I like Katie Williams’ take on the whole thing.
Hi Paul Thanks for this. I agree that these are very helpful. Williams point about enjoying writing and about post docs is absolutely spot on IMHO and the postdocs being able to write great papers from their PhDs.
ButI don’t think that she is altogether right in saying that “all you have to do is good work”. I see the REF process as much more political than that and some of this last REF attests to that. (Just ask Goldsmiths’ sociology). And some of the tips she offers do apply to all papers not just the 4*.
“In my view it would be a pretty disappointing social science school which skewed its entire effort towards the production of “outstanding” papers.”
Sadly that is the road we have gone down in BS/MGMT schools – where people work to a list and you try and make sure that you get four or three star publications off the list – even though the last REF didn’t use the List – to enforce this behaviour even beyond punishment, some places offer cash bonuses to staff to get these outstanding publications.
One of the outcomes of this is that it is killing off anything but the journal article – in our UOA – of 12,204 items submitted – off the top of my head maybe 50 were books and another 200 were book chapters.
Sounds like the Bs/MGMT community needs to have some more discussion?
Fascinating post. We ran a workshop for each of the Units of Assessment in our Faculty to discuss the difference between 1/2* versus 3/4* papers, with examples of each to discuss, and came to quite similar conclusions. One thing that fascinated me as we applied the REF criteria of originality, significance and rigour to each of the papers was that the difference between 1/2* and 3/4* papers often seemed to be dominated by the significance criterion, and it became apparent in some of these cases that where you have highly rigorous and original research, it may be possible to shift the ranking of your paper simply by re-framing it to draw out its international significance more clearly. Things like including a country (or worse region/district/city) in the title of a paper don’t help your case, but with a few carefully phrased sentences in your abstract, introduction and conclusion, you may well be able to make the case that the work you’ve done is significant enough to warrant a better grade. Of course this won’t always work – if the data isn’t that robust and can’t tell a significant story, then there’s nothing you can do to dress this up. But letting colleagues with more experience of the assessment criteria pre-review papers before submission could catch the odd paper that could benefit from this sort of re-framing, and be a very efficient way of boosting scores.
In our workshop, we also discussed the concern that REF would incentivise us to only write 3/4* papers, and we recognised the need to actually purposefully write 2* papers for specific audiences if we have a narrow but important contribution that needs to be read by a specific audience. Drawing out the international significance of such work may in fact dilute the relevance of the findings for its core audience and limit the societal impact of your work.
Yes I completely agree with both of these points, thanks for elaborating.
Pingback: Link Round-Up: Academic Journals and the Publishing Industry