Building a Poll Part 4: What to Ask About

There is one thing about polling that makes it incredibly frustrating for a pollster and great for journalists and other pollsters who seek to discredit your findings: there is no such thing as a perfect methodology.  There is absolutely nothing that you can do to ensure that your poll is perfect. Not too long ago, The American Prospect had an informative review of a recent book on the war on science. I'd advise that you take a look at it and think about it before we proceed.  Your poll will never be perfect - come to terms with this. Someone will always be able to complain that you should have asked X or should not have asked X, that the ordering of your questions introduced bias into the poll by creating incentives for certain responses, etc.   Some of these complaints are nothing more than standard trolling and attempts to discredit the findings of a poll as described in the article I linked to, but some of them are legitimate. So long as your poll falls into the range of best practices (which we're going to be describing in this blog), you're fine.

 How do we know what to ask about? 

Well, as I discussed in the first post of this series,

 WHAT - This is pretty straightforward. What is it you want to know from these five hundred people? Are you asking about their opinions on waste disposal? Are you curious about their opinions on a political issue or a candidate? Also, in addition to these opinions, what else do you want to know about them to help put these opinions in context? If you can answer these questions, then congratulations are in order. You have just determined the dependent and independent variables in your experiment. You can think of the opinions as the dependent variable, and the contextualizing information as the independent variable.

Let's start thinking through this.  In case the Wikipedia article on dependent and independent variables wasn't clear, here's an easier way to think about it.  You are trying to determine how the dependent variables change with respect to the independent variables.  So, for example, you want to see how support for the Republican Party changes as the age of the people in your poll change.  Another example would be if you wanted to know how support for attacking Iran changed with respect to party identification and income.   In the case of the former example, you are explaining Republican Party support in terms of age. In the case of the latter, you are explaining support for attacking Iran in terms of party identification and income.

So you know that there are two subdivisions in what you have to ask about: the dependent and the independent variables.   The independent variables are somewhat easier to figure out. Basically, you want to know the details of these respondents' lives.  You need data that will help you build a picture. So you ask about their gender, age, income, level of education, religious identification, etc.  The independent variables that you choose obviously vary based on the conditions of the environment in which you're polling, but the basic idea is that you get demographic data on these people.

Now comes the fun part: figuring out which dependent variables you want to use.  This becomes the fun part.  As a pollster, you're going to receive an incredible amount of information from your client about the project.  This is usually in the form of huge binders of information, but these days, smarter people are delivering soft copies.  This will tell you about the issues that they want to explore, the background on the campaign, etc.  It is your job to pore over these massive binders, PDF's, etc. and determine the relevant information that you should ask about.

Is there a good way to do this? There's no guaranteed silver bullet method, unfortunately.  Some of it will come down to your own good sense, and other parts of it will come directly from the client.  Working together, you will figure out what questions to ask about.  

Dirty D