Building a Poll

There is one thing about polling that makes it incredibly frustrating for a pollster and great for journalists and other pollsters who seek to discredit your findings: there is no such thing as a perfect methodology.  There is absolutely nothing that you can do to ensure that your poll is perfect. Not too long ago, The American Prospect had an informative review of a recent book on the war on science. I'd advise that you take a look at it and think about it before we proceed.  Your poll will never be perfect - come to terms with this. Someone will always be able to complain that you should have asked X or should not have asked X, that the ordering of your questions introduced bias into the poll by creating incentives for certain responses, etc.   Some of these complaints are nothing more than standard trolling and attempts to discredit the findings of a poll as described in the article I linked to, but some of them are legitimate. So long as your poll falls into the range of best practices (which we're going to be describing in this blog), you're fine.

 How do we know what to ask about? 

Well, as I discussed in the first post of this series,

 WHAT - This is pretty straightforward. What is it you want to know from these five hundred people? Are you asking about their opinions on waste disposal? Are you curious about their opinions on a political issue or a candidate? Also, in addition to these opinions, what else do you want to know about them to help put these opinions in context? If you can answer these questions, then congratulations are in order. You have just determined the dependent and independent variables in your experiment. You can think of the opinions as the dependent variable, and the contextualizing information as the independent variable.

Let's start thinking through this.  In case the Wikipedia article on dependent and independent variables wasn't clear, here's an easier way to think about it.  You are trying to determine how the dependent variables change with respect to the independent variables.  So, for example, you want to see how support for the Republican Party changes as the age of the people in your poll change.  Another example would be if you wanted to know how support for attacking Iran changed with respect to party identification and income.   In the case of the former example, you are explaining Republican Party support in terms of age. In the case of the latter, you are explaining support for attacking Iran in terms of party identification and income.

So you know that there are two subdivisions in what you have to ask about: the dependent and the independent variables.   The independent variables are somewhat easier to figure out. Basically, you want to know the details of these respondents' lives.  You need data that will help you build a picture. So you ask about their gender, age, income, level of education, religious identification, etc.  The independent variables that you choose obviously vary based on the conditions of the environment in which you're polling, but the basic idea is that you get demographic data on these people.

Now comes the fun part: figuring out which dependent variables you want to use.  This becomes the fun part.  As a pollster, you're going to receive an incredible amount of information from your client about the project.  This is usually in the form of huge binders of information, but these days, smarter people are delivering soft copies.  This will tell you about the issues that they want to explore, the background on the campaign, etc.  It is your job to pore over these massive binders, PDF's, etc. and determine the relevant information that you should ask about.

Is there a good way to do this? There's no guaranteed silver bullet method, unfortunately.  Some of it will come down to your own good sense, and other parts of it will come directly from the client.  Working together, you will figure out what questions to ask about.  

Dirty D

This has been a fun series to write so far, and I'm hoping that all of you are getting as much out of reading it as I am from writing it. In my few posts so far, we've stated a few assumptions and a few things for role-playing. I'd like to collect this all together and put it all in context.

In my last post, I set up a framework for thinking:

We now get further into the role-playing aspect of this series. Let's pretend that you're trying to design a polling strategy for a Democratic Congressional candidate running for an open seat in the state of Mississippi. It is going to be in time for the 2008 elections, and not a special election.  You are the campaign manager, and it's your job to figure out what you need to do with the polls.  You have no primary opponent, and are able to begin running against the Republican from day one.  In fact, your candidate had an exploratory committee up and going by March 07, and declared his intentions for the seat in June 07.  The Republican incumbent of fourteen years is retiring, and they're facing a primary of three candidates.  Their primary is not going to be messy and fun. It's going to have the party leadership step in and make their executive decisions, so you'll be running against someone by January 08.

Now, just to put this all in context, let's assume that you've been given the job because you're a master at making the trains run on time, and not necessarily the strongest on strategy.  Because of this, you're seeking advice from all kinds of people on what to do about polling.  And this blog will be the aggregated voice of all those people. Does this make sense?

Now, in the post prior to that, I decided that when we were discussing some completely hypothetical poll, we would keep the n-size at 500.

We now have two distinct conversations happening, and I want to make sure that everyone is clear on the difference.   In the first case, we'll be talking about building and using different kinds of polls, as well as interpreting them, in an applied context.  In the second case, we'll be talking about the abstract thinking that goes into different kinds of polls.

Clear? I hope so, because I'm looking forward to it.

Dirty D

We now get further into the role-playing aspect of this series. Let's pretend that you're trying to design a polling strategy for a Democratic Congressional candidate running for an open seat in the state of Mississippi. It is going to be in time for the 2008 elections, and not a special election.  You are the campaign manager, and it's your job to figure out what you need to do with the polls.  You have no primary opponent, and are able to begin running against the Republican from day one.  In fact, your candidate had an exploratory committee up and going by March 07, and declared his intentions for the seat in June 07.  The Republican incumbent of fourteen years is retiring, and they're facing a primary of three candidates.  Their primary is not going to be messy and fun. It's going to have the party leadership step in and make their executive decisions, so you'll be running against someone by January 08.

As we saw in my last post, there are five questions that anyone thinking through a poll will have to answer: what you want to know, from whom you want to know it, when you want to know it, why you want to know it and how you're going to find out.    The problem with these questions is that they're not discrete. Answering one is going to depend on answering the others, but at the same time, answering one is going to also allow you to answer the others.  In my way of thinking, the most important of these questions is the WHY question.  Let's assume that you read the longer descriptions of each of these questions and focus on the WHY question for now.

WHY - Now, this is the question that stumps almost everyone. Why are you asking these questions? If you can answer this, it will help you answer all of the other preceding ones. If you're asking questions to get a base case of data against which you're going to measure things, you're going to probably want to ask a lot of questions that you wouldn't want to ask if you were just doing a periodic update (which is what we call a "tracker" or a "tracking poll"). You're also going to want to ask different sets of people these different questions. Also, how does this poll fit into your larger strategic goals? Will acquiring these data help you update your telemarketing or door-knocking schedule? Will it help you figure out whom you should send fundraising mail?

The bottom line is that if you can't answer this for every single poll that you have commissioned, you didn't need to have commissioned that poll. That was money that was wasted on polling - you could have used it for your field operation, your media operation, etc.  So let's think about different reasons why people poll.

1. Baseline polls: These are the big ones, the juggernauts.  A baseline poll will generally be pretty big in length and will cover just about every conceivable topic. It's there to establish a base case of data which you will be later updating with tracking polls, so you want to have a question there for everything.  Why are these kind of polls useful?  Because they allow you to meaningfully measure the effects of your actions, the actions of your opponent and basically whatever other factors may affect your election.  This is the starting point for things - from here, things move either up or down based on the vagaries of the environment.   That's why you want to pack it as full of as many things as you can.
2. Testing: Before you spend an obscene amount of money being blackmailed by your consultants, it may be a good idea to test what they're telling you.  Actually, scratch that. It is always a good idea to test and validate what they're telling you. Call this Dirty D's Law of Politics: For every stupid idea X, there is an infinite number of consultants waiting to pad their bank accounts by telling that it's great idea on which you should spend all your money.   Use your basic common sense to filter out a bunch of the crap, and you're still left with a lot of ideas that you can use to advance your campaign.  Pick the few that seem like a good idea to you, and put them into a poll.  I should point out that this process is not foolproof, but it's better than nothing, provided that you accept its limitations. (We'll be discussing these limitations in later posts.)
3. Tracking:  When you have a base case of data, you want to see how you're doing relative to it. You want to see how your actions, the actions of your opponent and everything else that's going on is affecting your path to victory.   Every now and then, it's a good idea to figuratively call back and see how people are doing.  To stretch the journalism metaphor a bit, consider a tracking poll to be analogous to a call back or follow up interview by a journalist.
4. Reaction testing:  This is slightly different than regular testing and tracking.  Let's say that something momentous just happened that changed the political environment entirely.  Perhaps your opponent just found a thirteen year old girl to come forward and say that your candidate and his wife routinely sexually abused her in a Satanic sex cult. (Don't laugh - you should never be surprised at what can happen in a Congressional campaign, and this is not atypical.) Let's assume that the media ran with this story for days on end, and you've been besieged by calls.   This looks like a pretty good time to field a poll and see exactly how bad the damage is.  Similarly, if something momentous happens that doesn't pertain to your candidate, it might also be a good time to poll.  Basically, if you can take a deep breath, look at the environment and say that it's been qualitatively changed by some event X, it's a good idea to poll in its aftermath.
5. Special Projects:  As a single Congressional candidate, you're never likely to poll for anything outside the scope of finding the path to victory for yourself, but that doesn't mean that state parties, national parties, NGO's, etc. won't be doing this, and that you won't have access to these data.  You should understand the nature of special projects polling.  In these polls, people are asking questions that they wouldn't normally ask because they're trying to get data to build some kind of model that they can apply to a larger dataset.  Here's an example.  Let's say that the MS Democratic Party wants to measure how people get their news, whether they prefer nakedly partisan sources like The New York Post and Fox or objective journalism, and what degree of credibility they place in both.  Knowing how these preferences distribute over your various independent variables, like age, race, ethnicity, etc. will allow them to build a model that they can then use to predict the best methods to reach different demographics.  These data can be made available to a Congressional campaign like yours who will most likely be thrilled to have this kind of modelling.

These are basically the main reasons why people poll, but there are other reasons to poll. If you can't classify any poll as one of these, you should really, really rethink whether it's necessary.

Dirty D