the audit trail – a common omission from methods chapters

It’s really important to put an ‘audit trail’ into your methods chapter. Audit trail? Well, it doesn’t actually matter what you call it, it’s the section in the chapter where you give the examiner the nuts and bolts information about what you’ve done.

Providing an audit trail doesn’t mean you can ignore your philosophical stance and your choice of methodology. Of course you still have to write about this. But I see a lot of – read this as I see far too many – theses where that’s all people have done. There are pages and pages about epistemology and methodology and methods and nothing about what’s actually happened.

I can’t tell you exactly the number of ethnographies I’ve read – but it’s a lot – where I don’t know how many days the researcher has spent in the site, what they’ve looked at and what they were looking for, what documents they’ve collected and who they’ve formally talked to. This information partially dribbles out through the results. But if I’m not given this basic information at the start, I always approach the substantive and important results chapters with unhelpful questions and doubts in my mind. I am wondering to myself, “What’s the basis on which this description and this analysis is being made?”

BUT the consequence of leaving out the audit trail is generally greater than simply raising doubts in the examiner’s/my mind. Whenever a doctoral researcher has left out the audit trail, they are usually asked to turn around and put it in as a correction. And who needs that if it can be avoided?

The audit trail consists of two things:

(1) Information about the actual data you have generated
This is – how many people you’ve interviewed, how many days you spent in each site, how you constructed the survey, what kind of statistical approach you took, how many transcripts you are working with, how many blogs and posts, how many words in the corpus and so on. It’s the who, what, where, what kind, how long, and how many of the research. You can often do this in a table so it doesn’t take up more pages and pages.

(2) Information about how you’ve analysed the data
This might be the actual workings or it might be a sample of transcript, codes and themes and so on – whatever is appropriate for the methods you’ve chosen. This often goes somewhere other than the chapter and is referred to in the actual chapter proper.

Now the reason for providing an audit trail is not because your examiner doesn’t trust you. It’s not that your examiner wants to believe that what you’re claiming about your results is inaccurate or fudged in some way. The reason for providing an audit trail stems from the job that the examiners are asked to do.

PhD examiners have to make a judgment about the quality of the research. Is it ‘doctoral’? Examiners are not just looking for the original contribution but also whether you know how to do research and can be let loose on the world as an independent researcher, able to undertake unsupervised research that will be competent and scholarly. So they need “evidence” that you can do this.

At its most basic level, whether something is doctoral is a question of data quantity – has the person done enough work and got enough data to answer the question they’ve posed, using the method they’ve chosen. Depending on the question, this might be as small as one person or one text, or it might be as big as national data base. The doctoral researcher must tie the question, methodology, methods and data decisions and processes together. The thesis needs to both show and tell, and argue, the actual basis of the research.

At the next level, the examiners need to be able to follow what you’ve done. They/we/I can’t make a judgment about ‘doctorateness’ if all they/we/I am looking at is a set of results – if that’s all there is then they/we/I don’t actually know how you’ve arrived at them. If you don’t show how you’ve reached your conclusions then you are virtually saying to the examiner – “Trust me, I didn’t make it up, I was systematic and rigorous and I did look for things that didn’t fit as well as those that did, but I won’t show you what I did…”

It sometimes seems to me that researchers who do statistical work are more aware than others of the need to show that their calculations are error free and statistically defensible. But people working with interpretive approaches also have to be able to show what decisions they have made, why, how and the basis for them. In interpretive work, examiners are not looking for accuracy but they are looking out, for example, for the kinds of searching that has occurred in archives, the reasoning behind the interpretation of interview material and texts, the kind of critical interrogation undertaken.

And of course providing an audit trail doesn’t all have to be in the main methods chapter – you put the most important, the key, information in the chapter and then provide the rest as appendices. An over-detailed audit trail might well frustrate your examiner and you don’t want to do that either. That’s nearly as bad as leaving them uninformed. But not quite.

You have to get the balance right with audit, as with all things thesis. So it’s really important to get an idea of the general level of audit trail that’s provided in your discipline and then, as necessary, confine the rest of the workings and ‘evidence’ to an appendix where the examiner can follow it up.

About pat thomson

Pat Thomson is Professor of Education in the School of Education, The University of Nottingham, UK
This entry was posted in evidence, examination criteria, examiner, methodology, methods chapter and tagged , , . Bookmark the permalink.

11 Responses to the audit trail – a common omission from methods chapters

  1. Sue Gollifer says:

    I have just been reading about the audit trail, and so extremely useful to get your understanding and explanation of the nuts and bolts of the research. Great article that provides practical support on getting the balance for your specific discipline. Thanks.

    Like

  2. Silkie says:

    I found this post really useful. Didn’t think I would have the opportunity to explain the kind of information contained in the ‘audit trail’. Thank you!

    Like

  3. vinnycho says:

    Reblogged this on Datapulted and commented:
    Excellent thoughts on what an audit trail does to improve the trustworthiness (i.e. reliability and validity) of qualitative research.

    Like

  4. Pingback: the audit trail - a common omission from method...

  5. vivadrama says:

    This is great! I am instantly forwarding to my postgrad students….

    Like

  6. jovanevery1 says:

    The same can be said of research grant proposals. The practical details of what you plan to do: sample size, criteria for selection, data collection practice (days in field, # of interviews, etc), and techniques for data analysis are the MOST important part of the methods section of a proposal. (and for the methods section of dissertation, journal article, book, etc). They need to be contextualized in a broader debate about epistemology and methodology but at the end of the day, HOW you got your data is important to how we read the findings. And knowing these details is crucial to an adjudicator in a competition that’s handing out research funding to determine the feasibility of the study and the likelihood that you will achieve those fancy objectives.

    Like

  7. Gerard Ilott says:

    Oh Man! This is great. I am a 52yo “early career researcher” who actually managed large systems and audit trails in a past life. What a great way to explain the methodology chapter. First class.

    Like

  8. Thank you for the post. Very helpful and easy to understand.

    Like

  9. Very helpful. Thank you for making this concept completely clear. Now to add it to my dissertation.

    Like

  10. Shelley says:

    Yep, very helpful! I’ve been fretting about this and now have a common-sense way of approaching this in the Methods chapter. Thank you.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s