In this post Thesis Whisperer opens up the issues around METHODS – how to actually do the research we need to do on our joint paper on academic blogs.

The methods section, especially in a project that includes empirical research, serves an important function: to get your reader to trust what you are saying. The main purpose of the methods section is to make a convincing story of the (often) messy reality of conducting research.

And most research is messy.

We don’t talk about mess in research very often, but we should. The books you read and courses you take while doing your PhD can make the process of choosing and using a method look straightforward:

1) Choose what topic you want to research,
2) Ask some questions about it,
3) Design some experiments / conduct some interviews / collect some survey data, and then
4) Analyse it, to find
5) ‘the answers’.

The reality is more complicated. Research is an iterative process where we inch our way towards certainty – sometimes only partially achieving it at the end. The research that Pat and I are beginning about blogging is a good example of this iterative process in action.

We started out with some tentative ideas about what blogs were doing in academia and wrote an abstract. We formulated some questions; we didn’t know if they were any good but we had to start from somewhere.

The next problem was a doozy: how to create a good sample of blogs to study. What does ‘good’ mean? Well, it depends. In statistical studies a ‘good’ sample is one that is representative of a whole; in qualitative analysis it is less clear. Again this is an issue of trust. We must convince readers that we looked at enough academic blogs to be confident about any statements (knowledge claims) that we make about academic blogs as a whole (generalisability). But no one can say for sure how many blogs even exist; it’s impossible to even guess at how many of these blogs are run by academics. A representative sample is impossible.

I fretted over this problem for some time, in fact it became a road-block to even getting started until I started to read Helen Sword’s new book ‘Stylish Academic Writing’. Sword wanted to research the way academics write in order to be confident about advice that she gave on, for example, use of the first person in academic writing. Sword’s problem was similar to ours: there are many hundreds of thousands of published articles. Creating a representative sample, if not totally impossible, would be astoundingly difficult. As Sword writes:

“When I first embarked on the research which underpins this book I harboured a fantasy that I could map a coherent landscape of disciplinary styles, zooming in on specific regions and making informed pronouncements about their inhabitants: ‘anthropologists write like this; computer scientists write like that’. By the time I had assembled my initial data set however – one thousand peer reviewed articles from sixty six different journals in ten disciplines across the arts, sciences and social sciences – I realised that a panoptic overview of signature writing styles across the disciplines would be impossible”

Realising my task was, in fact, impossible was invigorating. If I couldn’t get a ‘good sample’ or even know what ‘good’ was, I could only try for a ‘good enough’. If I was to build a ‘good enough’ sample I just had to start somewhere and continue to build for as long as I could. Good enough would have to be as much about time and resources as it was about the object of study itself. The genre of academic paper is important in coming to this decision; in a ‘work in progress paper’ such an approach is much more acceptable than in other cases.

A common practice in blogging is to create what is called a ‘blog roll’; a list of blogs that the author has included on their site because they are friends, or the author thinks the blogs are worth visiting. This enabled me to use a modified ‘snowball recruitment method’: using blogs to recruit more blogs. I had a rich array of links to other academic advice blogs, as well as a significant number of PhD student blogs off my own blog, The Thesis Whisperer. By following those links outwards – and finding the blogs that were connected to these blogs (friends of friends if you will) I quickly built up a sample of around 100 blogs run by academics or research students.

The snowball method is a good way of accessing a community that might otherwise remain hidden, but it holds the danger of creating ‘filter bubbles’. Potentially there were many other academic blogs that were not strongly connected to my network. I might have ended up mapping only isolated communities that are not representative of academic blogging as a whole. To increase our confidence in the sampling method I needed to find other ‘jumping off’ points, not just those blogs linked in some way to my own site.

Luckily the Guardian Newspaper in the UK had recently launched a sub-site to house academic blogs on Policy, Leadership, student experience, research, professional development, marketing and communication, finance, technology and international issues. These blogs provided more jumping off points, as did similar sites such as the LSE lists of academics on Twitter.

This helped me with my filter bubble problem, but I still wasn’t really satisfied. Then Pat sent me a link to one of the ‘methods toolkits’ supported by the ESRC, “Using blog analysis” was a description of a project conducted by Helen Snee from Manchester University. Snee had a slightly different agenda to our project and therefore used a different sampling method. She wanted to explore the narratives young people use when talking about taking time off between secondary school and university. Snee used commonly used terms such as ‘gap year’ on google blog search to locate potential sources. Using this method I ferreted out yet more academic blogs. My list was now heading towards 200 at great speed.

Now to deal with all this information… hmm. Let me think about that and get back to you. I’d be interested if anyone has suggestions or comments to make about my selection strategy so far. Do you think I have established a good method for finding blogs? What might I do differently?