Voter File

Of course, once you have a file for a given state, the work is not over. Even keeping the file at its current state, let alone improving it, will be an ongoing process.

First of all, from the day it goes up, the data gets staler and staler. Ideally, every state in the union will be undergo the entire process frequently--especially for states with hot races in election years.

Aside from regular rejuvenation, however, a file can be improved upon through contact with reality. No matter how diligent the Secretary of State, certain problems on a file will slip through the cracks. People will move, die, or get convicted; phone numbers will change, or go bad; party registrations will be altered. All of these changes can be captured by a well-tuned field organization, and appended to the file, so that as election day gets closer the file can asymptotically approach perfection (this is a somewhat idealized picture; bear with me).

When volunteers go out and canvass neighborhoods or phone bank, they can verify if an address is attached to the right name or whether or not a phone number is good; they can also gather information that is simply unavailable from other sources, like a person's top issue priorities. All of this information is gathered, centralized, and scanned in, so that the state voter file is as up-to-the-minute as possible. In the past, this was done by hand (when it was done at all); now, the use of new technologies like computers, palm pilots and bar coding of responses has greatly increased the efficiency of doorknocking and other field techniques.

Your weekly roundup of what's been going on at Overdetermined.net. Enjoy!

• Blue Leader takes a look at some truly lame poll reporting from Politico

• Pluribus has a couple of posts about the statistical programming language R: an overview of a NY Times piece and a pointer to a tutorial. If you're looking to do any sort of data analysis, it's probably helpful to know R.

• Dirty D has a follow-up to his post last week about Howard Dean, this time reacting to the appointment of Tim Kaine as DNC Chair

• And Blue Leader takes a look at how databases are changing the face of journalism

Once you've gone through this process, you should have a list with millions of entries, each containing personal and consumer information--ideally for every registered voter, and all non-registered adults.  So what can you do with it? Plenty.

Once it's compiled, the data has to be accessed.  Various people can be granted different levels of access--making the whole file available to any volunteer would raise serious privacy concerns, not to mention possibly giving access to rival campaigns or, god forbid, the other party.  For low-level volunteers, this access can be extremely limited, while higher-level operatives can be granted more generous permissions.  Broader access can be granted through a web interface like the DNC's Votebuilder, RNC's Voter Vault, or Catalist's Q-tool.  Using some relatively simple Boolean logic, you can create lists of all the people in a state, district or precinct who share certain characteristics--for example, you might want to find all registered black voters under the age of 40.  With a certain (ever-diminishing) amount of inaccuracy, this is a trivial list to pull.

As you can imagine, this is extremely useful.  You can use these tools to do everything from create walk lists for your volunteers to pull samples for polls or blanket a state with direct mail.  Which is why these files are considered so valuable, and why making them is big business--with big consequences.

Cross-posted from Overdetermined.net. Find the latest entries in the series there!

Once the data is (yes, is, prescriptivists--I went there) in a standardized format, we move from the realm of "interesting" into "faintly creepy".  The information from Secretaries of State or state parties is generally pretty innocuous--name, address, maybe phone number or age.  The appended consumer data, on the other hand, is more unsettling.  There's nothing on there that would do real damage if anyone knew it--no credit card numbers, nothing that people could use to steal your identity--but it can be kind of strange to think who realizes that you own two dogs and a cat.

Most of this consumer data is gathered by for-profit companies, who then retail it to both the state parties and the for-profit companies that are creating these files (if you take a look at our resources page, InfoUSA is one such vendor).  They get their information anywhere they can--state licensing agencies (think it might be worthwhile for the McCain campaign to know who has a gun license?), magazine subscription lists, grocery store value card memberships...basically, if you have to fill out a form for it, somebody wants it, and will get it unless prevented by law. 

Moreover, based on this consumer information, it's possible to predict other characteristics (within limits, which I'll go into in a later post).  For example, the RNC might conduct a truly massive poll that measured all kinds of behavior--TV habits, income, type of location, and lots of other things besides.  Based on that poll, they might determine that there's a high correlation between a given cluster of characteristics and certain behaviors.  For instance: only a survey can tell you how much radio someone listens to.  But it's possible to know for everyone where they live, their age, and whether or not they own a boat.  If all males 54-65 who have boat licenses listen to Rush Limbaugh, it can be a good predictor. 

This use of consumer data is at least a partial definition of the oft-abused term "microtargeting" (this WaPo article, although overwrought, is a good introduction).  Rest assured I'll have more to say on the topic in the future; but this is the overview.  Stick around; tomorrow, I'll go into how this data gets used.

A quick roundup of what's been going on at Overdetermined.net. Enjoy!

• Blue Leader speculates on the RNC's new in-house think tank

• The latest installment of Building a Voter File discusses matching between lists without a persistent and unique ID. Hint: read the earlier entries first, it will make things much more comprehensible.

• Pluribus explains why he doesn't trust proprietary voting machines

• Dirty D talks about using census data as a pollster

• And of course, a little graph-related humor

As I mentioned yesterday, this is the first entry in another series from Overdetermined.net. Later entries in this series can be found there. Enjoy!

Karl Rove got your mom's phone number the other day.  Not directly, but through a mutual acquaintance--probably a secretary of state.  Both parties--and plenty of other organizations besides--are engaged in creating massive stores of information on voters all across the country.  This is their story.

In order to run any sort of political campaign, it’s crucial to know something about your voters.  Who in a given neighborhood is registered to vote? Who isn’t, but should be? Who falls into what demographic group? Polling can tell you something about people—even a small group can contain useful information about an entire demographic, thanks to the magic of sampling—but to get bulk information about who lives where and what they’re like, you need a full-fledged voter file. 

In a very basic outline, the process goes something like this: data is collected, combined with other sources, and put into a useful format.  This data can be analyzed, improved, added to, and eventually used to run a successful campaign (Nate Silver has a useful primer).  Today, we’re going to focus on the very beginning of this process.

One symptom of America’s unupgraded beta of democracy is our insistence on federalism, even when it makes our lives difficult, and the way we collect voter data is no exception.  Every state’s voter file contains different information, arranged in different ways, with different formats for presenting the information.  Better than nothing, but not very useful if you want to run a national campaign, or even append information to everyone in the country (as opposed to a particular state).  Before any of the interesting stuff can begin, the information needs to be in one single format.

First of all, the file needs to contain a unique identifier for each person.  The states are required to assign these IDs—thanks to the Help America Vote Act, or HAVA—but their identifiers are only unique statewide, and sometimes not even that (for example, it might have to be combined with the county a voter is registered in).  Only an ID assigned by the organization creating the file can be guaranteed to be unique (and if the file doesn’t contain a unique identifier, it’s extremely difficult and unreliable to verify people’s identities without tracking them via unique ID, thanks to inconsistencies in reporting various types of data).  If your name is John on one source of information, Johnny on another, and Jhon on yet a third, it’s a challenge to ensure that all this really refers to the same person. 

In addition to a simple list of who lives in the state, the voter file will usually contain other useful information--address, previous vote history, registration information, date of birth.  Of course, not all of these are available from every state.  Not every state keeps track of party registration, for instance.  Some states have detailed vote history, others record only a few elections.  But at the end of this process, a raw info-dump has been turned into something that can be combined with information from any other state.  Tomorrow: appending!