Betrayed By Polls

Let me start this post with a full disclosure. I am addicted to polls. I can't wait for new polling data to come out and when it does I take it all in for a few minutes and them immediately want more polling. I check the RCP Average multiple times a day. I follow the Gallup and Rasmussen Tracks everyday, and when Gallup has McCain doing worse than Rasmussen I rationalize in my head why Gallup polling stinks. Heck I even read commentary about polling. But the other day I realized something. Polls at this point in the game are completely meaningless, and when the election gets closer they still won't be all that useful.

How did I come to this conclusion? Well it all started the other day when McCain was having a bad polling day, then out of nowhere later in that very same day came Wisconsin University with their Big 10 Battleground polls, and low and behold there was McCain tied with Obama in Iowa! "Aha' I said to myself "I knew Iowa was going to be close, there couldn't have been that dramatic a shift in 4 years". Looking for a trend in Iowa I waited until the next day to see what other pollsters found. And wouldn't you know it but Survey USA had Obama up 11 in Iowa. I was mad and angry. Not because the poll showed McCain so far down but because I felt betrayed(That’s right I felt betrayed by a poll, that’s how you know that when the election is over you need to get outside more and retrieve your sanity). For a moment I tried to defend the polls like a battered spouse would. "It's not the polls fault. Big10 polls aren't even rated at538.com. Survey USA gets top marks, so just stick with the reliable polls and you will never be treated like this again."

So that’s what I did. I went to 538.com and pulled up their pollster ratings and found that among the top rated major polling firms were Rasmussen, Survey USA, Research 2000, Quinnipiac, and Mason Dixon. Since MD doesn't have state polls out yet regularly and R2000 is a DKos poll that received high ratings only because they polled the Democrat primaries so well (wackos polling wackos). I stuck with Quinn, Ras, and Survey and used their latest polls (but no pre-Sept polls). Here is what I found.


State/ SUSA/ Quinn/ Ras
Ohio/ M+4/ O+5 /M+3
Virginia/ O+4/ - / Tie
N. Mex/ O+8 / - / M+2
N.J./ - /O+3 /O+13

Now unfortunately we don't have too many more states that were polled by 2 or more of these 3 firms in the last couple of weeks but based on what we have it makes you scratch your head. Who's winning in Ohio? Virginia? Is it close in NM or NJ? Who knows. All I know is that the top rated pollsters from 538.com can't agree on any of those questions. Now I must say in Washington, Florida, and Georgia they have very similar results to each other, but that just makes it more difficult to decide to trust them or not.

Not completely convinced I was wasting my time with polling I checked to see if each of the above pollsters were at least steady within their own polls, and all to often I found that not to be true. For example Rasmussen had McCain up 5 in Florida and had the race a tie in Florida in a two week span! Is Florida a tossup like 2000 or slightly out of Democrat reach like 2004? Rasmussen couldn't even stick with one answer for more than a week.

My time and resources were running out in my attempt to vindicate the polls but I had a few more tricks up my sleeve. I could check where the polls were at this time in 2004.

A couple of examples from Mid September 2004:

Florida- ARG Kerry +1 Gallup Bush +9 - Actual Result Bush +5
Ohio - MD Bush +7 SVis Bush +12 - Actual Result Bush +2
PA - Qn Bush +4 ABC Bush +3 - Actual Result Kerry +3

National -USAGallup Bush+14 CBS Bush +9 - Acutal Bush +2.4

Mid September in 2004 and everyone was all over the place, the same is true now. And last time all different firms in all different states were wrong. The same could be true now.

Trying to find one final reason to believe I looked to the RCP Final Poll Average. Maybe if you looked hard enough at all the polls combined like RCP does you can find the truth. Now remember these numbers are from just before the election not mid September.

State / RCPAve / Result
Florida /B+ 0.6 /B+ 5
Ohio /B+ 2.1 /B+ 2
Iowa /B+ 0.3 /B+ 1
Wisconsin /B+ 0.9 /K+ 1
Colorado /B+ 5.2 /B+ 5
Michigan /K+ 3.5 /K+ 3
Penn /K+ 0.9 /K+ 3
Hawaii/B+ 0.9 /K+ 9

Ok finally we find some good results. RCP was dead on in OH, IA, CO, and MI. Off a bit in PA, and FL(enough to matter in close election). And picked the wrong winner in WI and HI. This is a strong result but completely dropping the ball in one state, being off in another, and not being dead on in a few more could make for a dramatic swing in the electoral college projections from reality this fall.

So what does this all mean and what did I finally convince myself of. First of all I will no longer take polls too seriously until late October. Second, I will rely more on the Averages at RCP and 528 than on any individual poll. Third, every time I see poll that is within 5 or 6 points one way or another I'll consider it a tie that could go either way. And Fourth, I won't stop looking at polls like a madman I will just remind myself that most of the polls will betray me