The recipe for good research

I’ve recently had the pleasure of working with NATS on a research programme aimed at discovering people’s attitudes to airspace modernisation and aviation in general. How do you design a robust survey to find out people’s views on a topic as complex and technical as aviation?

Our views on a variety of issues tend to be nuanced and not always perfectly consistent. This makes designing a survey a challenge as the way the question is worded can influence our response. For example, if we ask a question such as ‘Do you agree that planes are the best mode of transport?’ people are more likely to agree than if we ask ‘Do you agree or disagree that planes are the best mode of transport?’ Even then we are leading respondents by only mentioning one mode of transport – their brain might have temporarily forgotten that. Instead the best way to ask that question is to ask people ‘Which of the following do you think is the best mode of transport?’ and include options for planes as well as trains, cars and bicycles. That way if a majority still say that planes are the best mode of transport, we can be quite confident this is the case.

This is especially important when we’re asking about national issues, which have potential pros and cons on both sides. We want to make sure we’re providing our clients and the public with accurate and robust insights into public opinion. That’s why we’re signed up to the Market Research Society and British Polling Council rules; these ensure that our research is conducted fairly and transparently.

Once we’ve designed a clear and balanced survey, our next challenge is to find people to take part. These types of surveys are usually undertaken in one of three ways: face-to-face, by telephone and online. When you’re looking to survey a representative group of people, face-to-face is the most time consuming and expensive of these methods; it involves knocking on doors up and down the country until enough people take part (usually 1,000 or 2,000). Surveying people by telephone is easier but can also be quite time consuming, while surveying people online is the easiest approach. Each of these methods has its own form of ‘response bias’, in other words, there are always certain groups of people who are less likely to take part and therefore more likely to be underrepresented in the survey. In the case of face-to-face and telephone surveys, this tends to be younger people who are less likely to be at home or answer the phone. In the case of online surveys, this tends to be older people and people with disabilities who are less likely to be online and less likely to sign up to take surveys online. Over time, as most of the population gains internet access this proportion has decreased meaning that online surveys are more reliable than ever. At the same time, it is always important for us as researchers to ensure our surveys account for this potential bias through techniques such as setting quotas and ‘weighting’ responses so they are representative of different demographics including age, region and gender.

One of the most important elements of our research for NATS was that we wanted to speak to people who lived near some of the larger airports in the UK. What’s the best way to do this? Using postcode mapping tools, we established which postcodes lay within 20 miles of each of the airports we wanted to analyse. We then targeted people based on these postcodes so we could make sure we had at least 100 responses around each of eight major airports. As a result, we were able to see whether the views of people who are most likely to be affected by changes to airspace differ from those of the population as a whole.

So there you have it, the recipe for a good survey is: designing comprehensive and balanced questions, finding a good spread of people to take part, and making sure you have good representation from people who may be most affected by the issues at hand.