Once upon a time, when I worked in schools, early childhood teachers routinely issued young children with a ‘pen license’. A pen license was much sought after as it meant that a child could ‘advance’ to using a pen instead of a pencil. Using indelible ink meant that the child was able to write legibly in longhand. But legibility wasn’t enough, the child also had to be able to copy and compose text without making lots of mistakes that needed to be erased. Writing in pen meant the pupil had been deemed competent at basic writing tasks.

Of course, while schools issued rubrics about what counted as the standard for the pen license, different teachers did interpret the rules slightly differently. And different children learnt differently, so they didn’t all achieve the license at the same time. However, by and large, it seemed that most children got their pen license well before going on to ‘big school’ in Year 3.

Why am I telling you this? Well, the modern PhD works a bit like a pen license – the PhD is a license to practice research. Once you have the PhD you can apply for grants in your own right, and you can also teach other people about research.  And you can supervise doctoral researchers. … although contemporary universities and funding bodies often require newly minted PhDs to keep working with a more experienced researcher in all of these tasks. But in theory you know – you have your research license…

Now it’s not unhelpful to think about the thesis examiner as responsible for granting a license to research. Understanding this helps you to focus on the things that the examiner looks for as ‘evidence’ that they can licence you … 

A key place where examiners focus their licensing attention is on the chapter which outlines the design and conduct of the research. The methods chapter, as it is often called and as it appears in the thesis, is not a mindless ritual, something to be done just because it is the done thing. It is actually the place where you demonstrate that you know how to design and conduct research – not just this one project, but others in future. You have to show you understand and can use the principles of good research.

Yes… what counts as good research design and research conduct does vary from discipline to discipline and also by research tradition. So it’s good to start your thinking about licensing by noting that your examiners will have been selected because they are expert in the discipline and tradition you are using, and have been institutionally deemed capable of deciding whether you will be deemed capable and competent.

However, despite all disciplinary variations, there are three things that most examiners look for in a methods chapter. They look at 1. research design 2. data generation and 3. analysis.

So I’m going to look at each of these questions in turn, but a health warning. Do bear in mind that what I’m going to say is pan-disciplinary and across research traditions and isn’t comprehensive. You still need to think about what these questions mean for you and your particular research.

1. Is the research well designed?

Examiners look at the nature of the research question – is it answerable? Have key terms been explained in terms of this study? Are there any ambiguities in question wording that the researcher hasn’t taken account of?

The examiner will consider whether the methods chosen are fit for purpose – this whether they will produce data* that will allow the question to be answered in some way. Your choice of what or who is involved in the research (sometimes called the sample) are vital. Examiners will make a judgement about whether the data to be generated by this design will be sufficient  – will it be ‘good enough’. In other words, they will anticipate the nature, quality and quantity of data that your methods will produce. And examiners will pretty well always look at the proposed analytic approach to see what kinds of understandings it might produce – and how this relates to the ambitions expressed in the question. Examiners  also look at the potential for difficulties to arise in the research – these might be ethical or analytic. Examiners look not only for fit but to see that you understand what this design can and can’t do.

2. Are there any problems in what the researcher actually did to generate data*?

 Examiners are looking for processes where there aren’t blank spots – some things are where  the researcher ought to have seen as a problem but hasn’t. Inevitably the research will do some things and not others, and examiners want to know know that you know this. So they make sure for instance that survey questions have been well constructed, experiments are carefully designed, that interview questions aren‘t inappropriately restrictive and closed, that observations have been focused in line with the research question … Examiners look for consistency of approach, regardless of whether the research is an emergent process, highly structured or something in between. They are also looking for how the researcher puts their ethical undertakings into practice.

3. Has the data* been analysed thoroughly?

Now I’m going to devote another post to this last question, so I’ll only say here that examiners look to see that you have used a rigorous process which has been consistently applied. This might be a statistical approach which involves complex modelling – examiners look for example for definitional assumptions, choice of technique, accuracy in application and calculations, appropriate tests for significance and so on.  Or it might be a contemporary ethnographic approach which involves writing processes, from field notes through to descriptions and then theorisations – the examiner looks here for the explicit chain of conceptual development. Or it might be the more usual qualitative approach of building themes and coding – the examiner looks for the processes used in transcription, the ways in which inductive and deductive connections have been produced, the development of categorisations and ‘coverage’ of the raw data and so on. Or it might be a process of interpretation which connects together various kinds of primary sources – the examiner looks for what sources, how they have been used, omissions and un/tenable assumptions…

The examiner wants to know whether all of the data that was generated has been used, whether disconfirming data was taken account of and how. They also want to know how any conceptual framework and/or theorisation was developed as part of the analysis.

Then… The sum of these three questions leads the examiner to decide whether the researcher gets their license. In some research traditions, the examiner is understood to be looking for research which is both valid and reliable, perhaps also replicable – in other traditions the examiner looks for the trustworthiness of the research process and the credibility of interpretation.

So what does knowing the importance of the three questions mean for the doctoral researcher?

Well, it means that the thesis, and particularly the chapter outlining the research process, has to provide the examiner with sufficient detail to make judgments. Examiners want to see:

• An explanation about why the research was designed in the way it was. Why this research question is worded in just this way and then the what, why, who, when, how many, how often…
• A discussion about the ways in which you generated data, including any problems you encountered. Some of the more detailed points as well as samples of the actual data ‘types’ may be included as supplementary material.
• An audit trail. A lot of this material can be confined to appendices. ( Note that not all examiners like appendices. Check out their preferences if you can. Me, myself and I love a good appendix.) The examiner usually requires  a list of all of the data, both raw and treated and relevant details such as place, time, source etc. They mostly want to see samples of the analysis –in some research traditions the entire data base and all of the workings are provided for checking, while in others selected anonymised data are presented. Documentation of the decisions made during the research – about data generation and its processing – provide a map for the examiner which shows how the project was brought to the point where it was written as a thesis.

So this all means paying careful attention to the places in the thesis where you present your examiner with the information about your research.

What do examiners not want to see? Yes, this is a serious question. With an answer. 

Don’t give the examiner a text without sufficient methods information. Nor the reverse. You don’t have to bore the pants off the examiner with mind-numbing detail, so do discuss with your supervisor what examiners need access to immediately, and what they need to access in appendices or supplementary materials in order to see the complexity of your work.

And remember…thinking about these three questions means that you’re well on the way to becoming licensed. 

A caveat

*Data is a tricky term. I’ve starred it just so I can say that it is usually applied to numbers, images and words, and all combinations of them. Data can be generated from texts through work in libraries, through numbers generated in labs or in selected sites such as offices, schools, factories and hospitals.  But there is also debate about whether there can be any such thing as data, and post qualitative researchers now seek other ways to conceptualise research practice altogether. This is a blog post so I can’t do much more than just show that I know that this term – like just about every other one I’ve used above – are contentious!

Image credit: Rich Remoneron