September 12, 2003
I Will Never Understand Pollsters
The National Journal's Charles Cook writes:
Charlie Cook: ...After a Iraq War spike in President Bush's job approval ratings and the nation's right-direction numbers in late March and through the month of April, both measurements have settled back to roughly pre-war levels, notwithstanding the results of some rather dubious polls and even sketchier interpretation of recent data, not to name names....
According to Ipsos Public Affairs/Cook Political Report polling, 57 percent of the 1,561 registered voters in the two combined August surveys (margin of error +/- 2.5 percent) approved of the president's overall performance in office, with 39 percent disapproving. That is almost identical to the 56 percent approve, 39 percent disapprove among the 4,709 registered voters interviewed during the first quarter of this year (margin of error +/-1.5 percent).... All of these numbers are very similar to the 57 percent approve, 41 percent disapprove level in July.... [H]is approval rating jumped up in April, dropped five points in May and lost a bit more ground in June and July, then essentially leveled off.
But while Charlie Cook is writing this, I am staring at figures like:
The Ipsos-Reid/Cook polls do indeed show what Cook says they do: the light-colored triangles that are the Ipsos-Reid/Cook results in August do show no decline from July. But the curious thing is that the many other pollsters do show continued decline. Is every other pollster that much inferior, that Cook would bet on his results (given sampling error) rather than everybody else's? Does he really mean to dismiss the mass of other polls as "dubious polls... sketchier interpretation of recent data, not to name names"?
Apparently he does, for in the following week Cook rips into Zogby:
Charles Cook: ...Zogby... ostensibly... Bush's approval rating dropping... 45 percent... astonishing... if correct.... There is only one problem.... Zogby does not ask a job "approval" question... wording will inevitably result in a lower (and more newsworthy) job "approval" rating.... Zogby... sampling likely voters.... But the difference on this point is not who is interviewed so much as what questions are asked.
Yes, Zogby asks a different question, and he screens his sample in a different way, and the Zogby greyish-purple diamonds are usually near the bottom limit of the poll scatter--all this is true. But doesn't this miss the big point? Isn't the most interesting feature the continued decline--especially since Cook had just said the week before that the decline wasn't really there, and was produced by "dubious polls and even sketchier interpretation"?
And lo and behold, as more polls come in they confirm the declining July-August-September trend seen in Zogby. They invalidate the "levelling off" seen in--and only in Ipsos-Reid/Cook. From late August to early September we see a drop from 59 to 52 in Gallup, which has consistently been as high relative to the scatter of other polls as Zogby has been low. We see a July-September drop of 59 to 56 in Harris.
And we see an August-early September drop from 56 to 52 in Ipsos-Reid/Cook.
This leaves me not understanding what Charlie Cook thought he was doing at the beginning of September. Did he really trust his own results--subject to sampling error and model error--enough to believe them in contrast to Zogby... and Pew... and Newsweek.. and ARG... and ABC? I mean, you are trying to tell your readers about American politics, aren't you? And elementary statistical thinking tells you that if all the other pollsters see a trend that your poll doesn't then your poll is probably wrong, doesn't it? Isn't it embarrassing for Cook to have to admit that this week his own numbers confirm what he said last week were "dubious polls and sketchier interpretations of data"?
I will never understand pollsters.
Posted by DeLong at September 12, 2003 05:01 PM | TrackBack
Well, I'm no pollster, but my reading of the raw poll data is that we're about due for another terrorist attack, or another war. Only a true pollster can tell you which.
Posted by: Dick Durata on September 12, 2003 05:39 PMIIRC, Zogby's been on the money more than the other pollsters.
Posted by: Tom on September 12, 2003 07:17 PMAnd pollsters will most likely never understand economists, which is why pollsters stick to polls and economists should stick to economics.
Posted by: Sean on September 12, 2003 07:50 PM>>Well, I'm no pollster, but my reading of the raw poll data is that we're about due for another terrorist attack, or another war. Only a true pollster can tell you which>>
perfect
Posted by: richard on September 12, 2003 08:05 PMBrad,
Have you seen the latest Fox poll of Bush's job approval? It has Bush's approval rating at 58% up from 57% last month (and all the Fox polls since June show effectively no change of approval rating since then). Meanwhile the rest of the polls capture this decrease and even the normally very pro-Bush biased Gallup poll has decreased to 52%.
I'm almost sure that Fox is cooking their poll numbers.
Posted by: Bobby on September 12, 2003 08:48 PMThere's a good 3-or-so-percent margin of error on any given poll, so I don't necessarily think that FOX must be cooking their numbers. But if you check out www.pollingreport.com, it seems that FOX has released a Bush approval poll every two weeks for a long, long while -- except for two weeks ago. That is, there is a late August poll that seems to be mysteriously absent. Perhaps that poll captured the decrease, and this poll -- by chance -- didn't?
I have no inside knowledge of any wrongdoing, of course. Maybe there's a good excuse for this apparently missing poll?
Posted by: Anno-nymous on September 12, 2003 09:47 PMIsn't Cook a Republican pollster? That would explain a lot, if so. So many of these pollsters are in the spinning game when you get down to it.
But it would be nicer if General Rove is betting on Cook for now. Dick Durata certainly nailed that one.
No, Cook's not a Republican pollster.
Posted by: Sean on September 12, 2003 10:03 PMReminds me of the official estimates about the most polular peace walks before the war on Irak... I mean if the guy is scratching his head then he must be a random bystander, right? Gee, those people were so wrong and so unpatriotic, remember them? Oh, that's right, there were all crazy bleftists (besides a lot of regular American families, Israelies* -God forbids-, etc.) so it doesn't count. Sorry, Babylon as usual...
* I swear on my honor. Not worth much, but that's all I've got pretty much.
Posted by: Jean-Philippe Stijns on September 12, 2003 11:23 PMI don't know whether Fox deliberately cooks their poll data (I would hardly be shocked) -- but they do consistently show a bias of several points to the Right of most polls (as Harris and Pew show a similar mild bias to the Left). And even Fox now reports only a 46-44 edge in support of Bush's requested $87 billion for Iraq -- which means that Gallup's 51-46 reported opposition to it is probably correct. (What's REALLY interesting is the 12-15% of the voters who tell both pollsters that our presence in Iraq is "justified", but then aren't willing to pay the $87 billion absolute minimum required to make it work! Cognitive dissonance lives...)
On a more encouraging note, I've long suspected that many of that 70% of the voters who are described as thinking that Saddam had a role in 9/11 aren't mindlessly swallowing Bush's line, but are simply operating on the vague overall belief that he hates the US enough that he probably had SOME role in supporting it, but didn't play the commanding role. Sure enough, I just tonight noticed June and July polls by the "Program on International Policy Attitudes"
( http://www.pollingreport.com/iraq.htm ):
"Please select what you think is the best description of the relationship between the Iraqi government under Saddam Hussein and the terrorist group al Qaeda:
(1) There was no connection at all.
(2) A few al Qaeda individuals visited Iraq or had contact with Iraqi officials.
(3) Iraq gave substantial support to al Qaeda, but was not involved in the September 11th attacks.
(4) Iraq was directly involved in carrying out the September 11th attacks."
3/4 sample (MoE ± 3.5)
7/03 6/03
No connection at all 7 7
Limited contact 35 26
Substantial support,
but not involved in 9/11 33 36
Directly involved in 9/11 20 25
No answer 6 6
In short, the American people are not quite THAT ignorant (thank God).
Posted by: Bruce Moomaw on September 13, 2003 01:46 AMBushs popularity can and will decline in the polls
in the next 12 months. The REALITY is Bush will carry EVERY southern state NO MATTER what,more layoffs, more terror, stock market crash. You name it. Bush will carry the South and not have to spend a dime. That leaves $200 million to keep the Dems from running the table in the Mid West. They will punt the East coast and California. I can't make the case for the Dems current cast. Deans recent comment about Lott and Mississipi shows he already is conceding the South. Labor is inconsequential, and the Minority
voters today DON"T make the EFFORT like their parents.
Polls or no, these folks are a fiasco from the environment to the economy to foreign relations.
Posted by: lise on September 13, 2003 08:36 AMMy suggestion is that you don't pay attention to the "approval" numbers. Pay attention to the reelect numbers. Those numbers are much worse and much more relevant. "Approve" all you want - but the bottom line is "would you vote for him?"
Hmmmm?
Posted by: Ian Welsh on September 13, 2003 09:02 AMBrad:
I've worked with pollsters. They are generally extremely self important hacks who get very invested in whatever version of reality they are pushing. I don't pay much attention to Cook, but from the times I've caught him on C-Span he comes across like a typical inside the Beltway type. That is, part of the political/media class that has been telling us how popular Bush has continued to be, even though it is doubtful he personally was ever really all that popular. If Bush does indeed go down the drain next year, fellows like Cook will be the last to acknowledge it.
None of this compares to CNN Bill Schneider's declaration prior to the '96 election that Clinton's goose was cooked... on the basis of a 2% drop in a poll with error limits +/-3%.
Then there was a Gallup poll in which pollers asked questions to contrast the moral character of Clinton and Bush. But when one examined the questions actually used-- which were not publicized and not readily available to the public-- one discovered they weren't parallel. To oversimplify, people were asked whether they approved of the president's affair with Monica and whether they approved of Bush's moral character. This before Harken, Aloha, etc. had made page A13.
Needless to say, polls need to be taken with a grain of salt.
Posted by: Charles on September 14, 2003 07:36 PM