July 12, 2004

Approval Polls

The most remarkable thing about presidential approval polls is the ten-point spread among different polls. You'd think the pollsters would have figured out by now whose methodology is biased. You'd think reporters would by now have learned to say that poll X is "usually 4 points below the average poll." But they haven't:

Posted by DeLong at July 12, 2004 09:15 AM | TrackBack | | Other weblogs commenting on this post
Comments

I've always wondered why there are "Republican" and "Democrat" pollsters. They look at the same data and come up with very different responses depending on whether they are paid by the Rs or the Ds. Doesn't that always make the analysis suspect?

Posted by: budgetwonk on July 12, 2004 09:21 AM

____

Brad
I was thinking about that very same thing a couple of days ago while staring at the Pollkatz chart you show. I have to laugh that all of those pollsters keep saying that there surveys are within a =/- 3% margin or so, and yet if that were true the poll band would be less and the location of any one poll within that band would be random. It just proves once again how much one must know in order to seek knowledge.

Posted by: pamur on July 12, 2004 09:26 AM

____

My understanding is that the different pollsters adjust the data in different ways. This is not necessarily a bad thing, but it means that poll results should be stated as conditionals rather than as unadored declaratives-- i.e., "If 'Hypothesis X' is true, then the percent of Americans who approve of Bush's policies is xxx".

Posted by: Matt on July 12, 2004 09:33 AM

____

I would think that every pollster worth the name would want to always obtain representative samples and remove all systematic bias' from his work. Afterall, this would be great for business.

However, with today's electorate, this is easier said than done and the problems for pollsters are getting worse. (Maybe that's why they get the big money!)

Here are a couple of problems that torment every pollster today:
1) Hang-ups - many people today, when reached at their residence, won't give a pollster the time of day. How many hang-ups does a pollster have to go through to get to the objective sample size and what do the numbers of Hang-ups do to the representativeness of the sample? No one really knows.

2) Cell phones - many people these days don't have a wired phone and rely entirely on their cell phone to save money. Pollsters don't call cell phones. What does this omission do to the representativeness of a sample? No one knows.

And the woes of pollsters only grow worse.

Posted by: Lawrence on July 12, 2004 09:56 AM

____

The "chopped down Christmas Tree" says a few things:

1. Bush only gets increases in approval from media events - 911, the invasion, getting Saddam - otherwise his trend is almost always down.

2. The erosion of Bush support isn't media driven, the only reverse spoke is Fallujah/Abu Ghraib/911 - otherwise it is ground level reaction to his policies and their effects.

3. He's almost certainly going to lose the election, since he cannot go negative, must pander to his base late in the cycle, and is at the point where an attempt at "Osamatober" will look like manipulation.

Posted by: Stirling Newberry on July 12, 2004 09:58 AM

____

Quoting without reference? Doesn't that count as an academic infraction?

Chart is from

http://www.pollkatz.homestead.com/

who also offers this comment:

http://www.pollkatz.homestead.com/#anchor_67

"30 June 2004

I, like many others, have long suspected that some pollsters may have their thumb on the scale, favoring Bush, in their polls. This would be foolish, of course, because the real bread and butter for the pollsters is in market research, where accuracy is more important than politics. Even so, that was my suspicion.

I was mistaken. I finally rolled up my sleeves and tested the hypothesis directly; not only were none of the results significant, none of them even suggested the swings I expected.

Pollingreport.com posts several "approval" polls from Bill Clinton's second term. I used this data to construct a "Clinton Index," similar to my Bush Index. To my wonder and amazement, I discovered that the rankings of the individual pollsters were not much different: Fox and Zogby at the extremes, everybody else clustered around one or two points. Most important, none of the ten pollsters in the database exhibited a significant, or even conspicuous, swing from Clinton low to Bush high (or vice versa). I'm convinced. The pollsters are honest, or at least deserving of the benefit of a doubt."

Posted by: ogmb on July 12, 2004 10:53 AM

____

ogmb--Doesn't that make it even stranger that the analysis of the data provided by Republican an Democratic pollsters always seems to favor their candidate, issue, etc.?

Posted by: budgetwonk on July 12, 2004 11:07 AM

____

Budgetwonk,

My guess/impression is that the politically affiliated pollsters aren't entirely interested in objective results. They are being paid by candidates or parties after all.

Favorable polls have positive results. They help raise money, they encourage the base, they may induce bandwagon effects or dishearten the opposition. That doesn't mean these people are overtly dishonest, but it may mean that given a choice of reasonable methods, adjustments, etc. they choose the ones most favorable to their sponsor.

Posted by: Bernard Yomtov on July 12, 2004 11:25 AM

____

It's worth noting that the person who compiles these poll results and generates these nifty summary graphs, namely Stuart Thiel (aka "Pollkatz", see http://www.pollkatz.homestead.com), has done substantial analysis of the bias of the polls he compiles. Specifically, he compared them to the metapoll point spread compiled by Ipsos-Reid Research. Here's his results:

WHICH POLLSTERS LIKE BUSH THE BEST?
(updated 6/30/2004)


Qualitatively:

---------------------------------------------
Statistically significant pro-Bush bias compared to Ipsos-Reid:

FOX News, NBC News-Wall Street Journal (NBC-WSJ), Gallup, and Pew Research
---------------------------------------------
Statistically insignificant pro- or anti-Bush bias compared to Ipsos-Reid:

ABC News-Washington Post (ABC-WAPO), CNN-Time, CBS News, Harris, Technometrica Institute of Policy and Politics (TIPP), New York Times (NYT), American Research Group (ARG)
---------------------------------------------
Statistically significant anti-Bush bias compared to Ipsos-Reid:

Investors' Business Daily (IBD) and Zogby
---------------------------------------------

Quantitatively [and I apologize I can't get the table columns to align nicely]:

Pollster______Mean Difference____t-ratio (t > 2.0
____________from Ipsos-Reid____or < -2.0 is
____________spread since_______statistically sig.)
____________Sept. 11, 2001_______________________

FOX_________+6.933_____________+5.093____________
NBC-WSJ_____+5.267_____________+2.830____________
GALLUP______+4.118_____________+3.362____________
PEW_________+4.053_____________+2.252____________
ABC-WAPO___+3.085_____________+1.880____________
CNN-TIME____+2.230_____________+1.049____________
CBS_________+1.944_____________+0.679____________
HARRIS______+1.794_____________+1.121____________
NEWSWEEK___+1.171_____________+0.612____________
TIPP_________+0.561_____________+0.221____________
IPSOS-REID__ 0.000_____________not applicable____
NYT_________-1.271_____________-0.403____________
ARG_________-3.093_____________-1.847____________
IBD_________-6.174_____________-5.644____________
ZOGBY_______-7.483_____________-5.383____________

RANGE (between highest and lowest) 14.416

[Source: http://www.pollkatz.homestead.com/files/bushindexprobushtable.htm]


Seeing these statistically significant pro- and anti-Bush biases among the various polls, you might expect to see a lot of the pollsters with statistically significant pro-Bush biases used to have statistically significant anti-Clinton biases in the 90's and vice versa. Mr. Thiel, however, claims he can't find any such statistically significant shifts:

*****************************
Regarding poll bias
30 June 2004
[Source: http://www.pollkatz.homestead.com/#anchor_67]

I, like many others, have long suspected that some pollsters may have their thumb on the scale, favoring Bush, in their polls. This would be foolish, of course, because the real bread and butter for the pollsters is in market research, where accuracy is more important than politics. Even so, that was my suspicion.

I was mistaken. I finally rolled up my sleeves and tested the hypothesis directly; not only were none of the results significant, none of them even suggested the swings I expected.

Pollingreport.com posts several "approval" polls from Bill Clinton's second term. I used this data to construct a "Clinton Index," similar to my Bush Index. To my wonder and amazement, I discovered that the rankings of the individual pollsters were not much different: Fox and Zogby at the extremes, everybody else clustered around one or two points. Most important, none of the ten pollsters in the database exhibited a significant, or even conspicuous, swing from Clinton low to Bush high (or vice versa). I'm convinced. The pollsters are honest, or at least deserving of the benefit of a doubt.
***************************

Posted by: Bill on July 12, 2004 11:32 AM

____

In fact, what was intersting about Pollkatz's analysis is that
(pretty much) the exact same polls had the same bias for or
against the incumbent. This does seem to argue for a bias
due to the form of the actual polling question; I would think
that a selection bias would affect results for Clinton vs. Bush.

Posted by: Matt Newman on July 12, 2004 11:48 AM

____

Budgetwonk, I can't answer this question because I'm not Pollkatz and I didn't come to that conclusion. But reading his disclaimer again, along with his partisan scores, I would interpret his conclusions as: There are partisan pollsters at either end of the spectrum, and there is a cluster in the middle whose foremost interest is representativeness rather than partisanship.

http://www.pollkatz.homestead.com/files/bushindexprobushtable.htm

Posted by: ogmb on July 12, 2004 11:49 AM

____

Brad,

If you focus a magnifying glass on a tiny piece of that graph, you'll see things through the eye of a typical political reporter (like Botox Woodruff, whom I'm watching now). They think each different dot actually shows a change in the fundamentals of Bush's approvals, and that Bush's polls look like a bunch of choppy waves instead of the straight line you'd get by run a simple regression through one of those four sets of points in the graph above.

Posted by: Bobby on July 12, 2004 01:15 PM

____

Brad,

If you focus a magnifying glass on a tiny piece of that graph, you'll see things through the eye of a typical political reporter (like Botox Woodruff, whom I'm watching now). They think each different dot actually shows a change in the fundamentals of Bush's approvals, and that Bush's polls look like a bunch of choppy waves instead of the straight line you'd get by running a simple regression through one of those four sets of points in the graph above.

Posted by: Bobby on July 12, 2004 01:16 PM

____

BTW, the phrase "Osamatober" should become a common and well-known one if Democrats want to counter Bush's manipulating of the election this Fall.

Posted by: Bobby on July 12, 2004 01:22 PM

____

One interesting thing I noticed about that graph is that there's finally a hint that Bush's ratings may be bottoming out. For the past three years, it's been an almost constant negative drop. In just the last month or two, there finally seems to be a bit of flattening.

Posted by: PaulB on July 12, 2004 02:13 PM

____

Since "approval" is a subjective concept, unlike, say, voting, it's entirely fair to expect that somewhat different ways of asking the question will, as observed, produce somewhat different answers. If you instead compare a *given* poll's deviation from the bulk, moving-boxcar-averaged z-score for all of the polls, you'll find that most of the polls are relatively stable with respect to the Z-score baseline. Some are jumpier, like Zogby, and som show wierd fits and starts, like Fox.

So the polls are not accurate (how could they be, given the subjectivity of "job approval", but many of them are quite precise. I see no mystery here.

Posted by: Malapterurus on July 12, 2004 04:24 PM

____

Pollkatz shows that Bush starts from spikes, then always goes down. Anyone working politics in the White House has been long aware, therefore, that they are working for a loser. The spikes are: (1) the 2000 election [which is not on this graph] (2) 9/11, (3) begin Iraq invasion, (4) find Saddam in hole. Those last three are, I think, the spikes on this graph. To win the election, therefore, the lesson for the White House is obvious.

Posted by: Lee A. on July 12, 2004 05:06 PM

____

YOU'd think two things
1)the pollsters would have figured out by now whose methodology is biased.

How ? The moment of truth comes once every 4 years. Data on how polls are different is easy to come by. Data on who is right and who is wrong are rare. In particular, I think, based on 2000 that there is a huge spread between prez horse race polls of "likely voters" with Gallup good for Bush. I have not seen a definition of "likely voters" and I don't think it is a simple filter like answer to how likely are you to vote ("100% certain" -> likely, "very likely" -> unlikely). This can be checked only on election day (and usually isn't checked even then though they could do it).

How can we tell that pollster A has a biased view of "approval of Bush" when we have no objective way to measure or even operationionally define true "approval of Bush" ?


2)reporters would by now have learned to say that poll X is "usually 4 points below the average poll."

The second thought is just another case of reporters being dataphobic. The weird thing about reports of polls in newspapers is that the poll is always compared to the older poll by the same agency. The comparisons are made by the pollsters. Reporters just report. doing some work like pollingreport or Dr Pollkatz is beneath them or something. I guess the weird thing is not that reporters don't bother with this but that bloggers do.

Posted by: Robert Waldmann on July 12, 2004 05:26 PM

____

PaulB, i happen to agree with you that Bush must be at or near his bottom. I make his absolute bottom 39%, which was the number he polled in the one hypothetical kerry/mccain v. bush/cheney (52-39, iirc)....

bill and ogmb, thanks for that info about pollkatz's analysis; very helpful....

Posted by: howard on July 12, 2004 05:40 PM

____

Wow, what a fine thread of comments. My sense of this, which is a hobby horse of mine, is that the issue isn't one of bias, but methodology. Polling that depends on telephones increasingly samples a population that doesn't mirror the general population--for reasons Lawrence cited way upthread.

There's also something else at play, which I can't exactly define. Like Stirling, I'm confident that Kerry is going to win and win big. Do the math and it seems impossible for polling to be so close, for three reasons:

1. Of the voters who voted Gore in 2000, fewer will vote Bush this year than vice versa.

2. Democrats are more motivated to vote for Kerry this year than in 2000, the Bush voters less motivated for Bush. Nader is likely to be no factor; certainly he will be a smaller factor.

3. More Americans are likely to vote this year than in 2000, because it is perceived as a far more consequential election. Since women's suffrage, it has been universally true that the more voters there were, the more likely they were to vote Dem.

And despite these facts, the polling is CLOSER than it was in 2000. I don't know what you call this discrepancy, but it's there.

Posted by: Jeff on July 12, 2004 05:46 PM

____

Compare and contrast rather scattershot and random approval pattern before 9/11 and the last couple of months. The similarities are remarkable. This is the well known Bush "drift effect". Rather non-inspiring approval ratings prior to the event when "everything changed", even lower now. Wonder why that is?

Posted by: bobbyp on July 12, 2004 05:49 PM

____

One of the interesting things about Nader is that, in all recent polls I've seen that compare the 2-way and 3-way races, Nader takes as much or nearly as much from Bush as he does from Kerry. In other words, the Kerry-Bush spread is nearly the same regardless of whether Nader is in the race. This is true acoss several polls. We can infer from this that the only remaining Naderites are the FU, NOTA! crowd, not the raging left. These people would apparently be as likely to vote for John Anderson or Pat Buchanan as for Nader.

Sad.

Posted by: Malapterurus on July 12, 2004 05:56 PM

____

"To win the election, therefore, the lesson for the White House is obvious."

Cutting off the life support for Old Ronnie didn't work though.

Posted by: ogmb on July 12, 2004 05:57 PM

____

The graph is great, and would be a great piece of evidence for statistical sampling error in a a first-year statistics class. The variance seems constant -- no heteroskedaticity (nor one would expect it). One is pleased by the trend.

The pundits worry about the 50% electoral threshold, as well they should. If Bush was trending fromn 90 percent to 55 percent, what difference would it make to real life outcomes. We are now in the range where the sampling errors, selection bias and errors in variables matter. Let's hope that the trend maintains itself, and that by September they won't matter.

It would be nice to know from a practical and theoretical standpoint, what Bush's baseline number is. 35%, 38%, 42%?

Posted by: Knut Wicksell on July 12, 2004 07:28 PM

____

It would help if pollsters published their questions. (They do, somewhere, but not to be easy to get at.) One of the important wordings should be, not "Who do you intend to vote for?" (which requires the "wasted vote" calculation - OK as a choice, but not as good for investigative purposes), but, "Who would you like to see as President/whatever?" That would help us see how many people really like independents.

Posted by: Neil on July 12, 2004 07:47 PM

____

The best opinion poll is the Presidential Future on Iowa Electronic Markets.

After months of lagging by about 10 cents, the Kerry/Edwards (winner take all) presidential future is now trading at the same price as the Bush future.

It is too close to call, and Nader may represent the balance.

Posted by: John on July 13, 2004 02:03 AM

____

Thanks to all for all the attention, to BDeL for the plug, and to Bobby for explaining my position. Regarding methodology, here's an e-mail I got from someone at one of the major pollsters:

* * *
2. Low undecideds are most closely related to "follow-up probes"--do interviewers ask undecideds, "Well, which way do you lean?"--and the quality of the interviewing staff in general--well-educated, well-trained interviewers tend to weasel an answer out of almost every respondent, while lazy, ill-trained interviewers tend to take "don't know" for an answer and move on to the next question.
3. Undecideds who are persuaded to take a position are more likely to be harboring negative than positive feelings. They are being polite when they say, "Don't know." When you push them, most often they say, "OK, negative I guess."
* * *
He goes on to discuss which pollsters are reputed to prod the undecideds and how that helps account for the perceived biases. (Sorry, I can't say who's who, except that generally they're consistent with the estimates.) This is reassuring, not only because of the political bias question, but because it supports my bedrock hypothesis that each pollster's methodology is consistent month in and month out. Without that, the data is garbage.

Cheers --

Professor P.

Posted by: professor pollkatz on July 13, 2004 11:02 AM

____

The Iowa Electronic Market was an interesting indicator before it became widely publicized. It failed - badly - in 2000. It is cheap to enter, and it would not surprise me at all if a significant number of the traders simply had partisan attachments to seeing their team win - e.g. bid their guy up - rather than particularly caring about whether they gained or lost $100. Given the self-selected nature of the market participants and the potential for manipulation, why would it be a better indicator than nationwide polls?

Marc

Posted by: Marc on July 13, 2004 02:26 PM

____

7172 You can buy viagra from this site :http://www.ed.greatnow.com

Posted by: Viagra on August 7, 2004 07:52 PM

____

2272 Why is Texas holdem so darn popular all the sudden?

http://www.texas-holdem.greatnow.com

Posted by: texas holdem online on August 9, 2004 12:15 PM

____

6236 ok you can play online poker at this address : http://www.play-online-poker.greatnow.com

Posted by: online poker on August 10, 2004 01:32 PM

____

1116 Get your online poker fix at http://www.onlinepoker-dot.com

Posted by: poker on August 15, 2004 02:07 PM

____

8402 black jack is hot hot hot! get your blackjack at http://www.blackjack-dot.com

Posted by: blackjack on August 16, 2004 05:03 PM

____

Post a comment
















__