August 02, 2003

Would Somebody Please Explain to Me...

Would somebody please explain to me how it can be that Fox polls routinely have George W. Bush's approval-minus-disapproval rating twenty percentage points higher than Zogby? I mean, this isn't rocket science: this is polling. If I wanted to generate such huge differences--differences that dwarf sampling standard deviations at least five-fold--I wouldn't know how to go about it. How do they do it?

Posted by DeLong at August 2, 2003 08:09 PM | TrackBack

Comments

Uh, Brad, you're being disingenuous.

One pollster's "disapproval rating" is not the same thing as another's, any more than my mother's yellow cake, made with egg yolks, is the same thing as the lady down the street's, made with egg whites and food coloring, or uranium hexafluoride, or whatever it is she uses to get that robust color.

Unless you tell us what questions these two organizations asked, and how they converted the numbers of various replies into indexes of approval and disapproval, you haven't told us anything. You're just spouting.

Posted by: David Lloyd-Jones on August 2, 2003 09:03 PM

Well, assuming they don't just lie, something I wouldn't put past them - a combination of leading questions and skewing the sample will probably do it.

I'd love to see their questions and their methodology.

Posted by: Ian Welsh on August 2, 2003 09:03 PM

Well, assuming they don't just lie, something I wouldn't put past them - a combination of leading questions and skewing the sample will probably do it.

I'd love to see their questions and their methodology.

Posted by: Ian Welsh on August 2, 2003 09:08 PM

Yeah. Well a cake in your mother's house and a cake down the street are both cakes. We're not talking the difference between cake and cake here: we're taking the difference between mousakka and tiramisu...

Posted by: Brad DeLong on August 2, 2003 09:11 PM

Three big possibilities, any, all, or none of which might be what's up:

1) The question asked. For instance, support for 'the President' might be higher than for 'President Bush', which might in turn be high than for 'George Bush'.

2) Sampling. Zogby I know samples 'likely' voters, while most other organisations sample 'registered' voters. This can change outcomes, often unpredictably. There are other ways to mess with this too, more subtly. Call during the day, for instance, and you'll get housewives. Call in the evening, and you'll get working people.

3) Weighting. This is the real demon, and they'll never tell you how they do it (or even if they do it), so it's hard to check. It's generally accepted practice to over-weight groups that are under-represented in a random sample, so that they match the universe from which you are sampling. This keeps the demographic characteristics of your sample stable from wave to wave (i.e. even if you had 45% women this wave and 55% women last wave, in each case they would be weighted to represent 50% of the population, so wave-on-wave fluctuations would not be attributable to sample flux). The problem is what you're weighting: are you weighting your sample for age and gender? Or also income and education? Race? Party identification? And what are you basing those weights on? The general population? Last election's voter list? The insiduous thing is that it would be entirely possible to try all of these configurations, and pick the one that most closely matched whatever outcome you were expecting. Nobody would be dishonest enough to say "I want my poll to show that Bush is well-regarded, let's mess with the weights." But it can certainly happen that the 'expected' outcome you're trying to match is informed not only by real expectations, but also by ideological blinders. Yielding a biased result to the poll.

One thing that would be interesting to examine would be whether Fox polls tended to give Clinton higher or lower ratings. If they were higher, than maybe Fox's tool is just skewed towards the incumbent. If they tended to rate Clinton lower, than for one or more of the above reasons, Fox's tool would seem to be ideologically skewed.

Posted by: Andrew Edwards on August 2, 2003 09:22 PM

Questionnaires, sample sizes, and past results for several Bush approval rating polls can be found at http://www.pollingreport.com/BushFav.htm

Here's the goods on Zogby and Fox in particular.

Zogby
N=1004 likely voters nationwide, error +/-3.2%.
Question: "Please tell me if your overall opinion of each of the following people is very favorable, somewhat favorable, somewhat unfavorable, very unfavorable, or you are not familiar enough to form an opinion. George W. Bush..."

Fox/News Dynamics
N=900 registered voters nationwide, error +/- 3.0%.
Question: "I'm going to read the names of some political figures. Please tell me whether you have a generally favorable or unfavorable opinion of each one. If you've never heard of someone, please just say so. George W. Bush...."

Posted by: Curtiss Leung on August 2, 2003 09:22 PM

Part of the effect might be due to responders telling the pollster what he or she thinks the pollster wants to hear.

If Fox identifies itself at the outset, its well-known pro-GOP and pro-Bush bias might cause some on-the-fence types to respond with lower unfavorables and higher favorables for Bush.

Posted by: Jonathan on August 2, 2003 09:23 PM

Jonathan has a good point for a fourth option to my post, although in my opinion that would be pretty darned unprofessional, and I'd like to give the Fox pollsters the benefit of the doubt.

Posted by: Andrew Edwards on August 2, 2003 09:27 PM

I've been monitoring the "www.pollingreport" site for years (polls were one of my specialties in college), and the impression I get is that Fox is the only prominent poll with a distinct conservative bias. The Harris and Pew polls seem to have a liberal one -- in all cases, several percent -- but I haven't been able to identify any coherent bias in any of the others.

Posted by: Bruce Moomaw on August 2, 2003 11:14 PM

I just noticed what you were really asking, and the answer is extremely simple: Zogby lumps into the "negative" category people who rate Bush as both "poor" AND "Fair". Fox and most other pollsters, by contrast, ask the simple question: do you approve of the job so-and-so is doing? -- and when asked that, about half the people who rate so-and-so as "fair" will on balance give him a nod of approval. For this reason, Zogby's officially declared "negative" ratings on absolutely everybody are consistently much higher than those in virtually every other poll.

Posted by: Bruce Moomaw on August 2, 2003 11:21 PM

Brad writes "Yeah. Well a cake in your mother's house and a cake down the street are both cakes. We're not talking the difference between cake and cake here: we're taking the difference between mousakka and tiramisu..."

No, Brad. Both cakes.

Apparrently polling is all Greek to you, but it is a commonplace about surveys that you can get wildly different sets of replies out of comparable samples of people just by changing the order of the questions you ask. "Do you think wild cats in the street are a menace to your children? Do you think the government should gas stray cats?" will get at least a 20% difference in replies from the same two questions reversed, " "Do you think the government should gas stray cats...? Do you think stray cats are a menace...?"

Zogby/Fox differentials are likely to be of the same order. It's just part of the crudity of polling technology.

Posted by: David Lloyd-Jones on August 2, 2003 11:25 PM

How do Fox and Zogby handle non-responses? What are their respective rates of response? One would think that professional polling organizations would know how to adjust for non-response. Perhaps this explains a systematic difference between their results.

I donít understand why some groups get under represented in the random sample unless there is a defect in the sampling frame. Why not use stratified sampling or unequal probability sampling if there is a problem with the sampling frame?

Posted by: A. Zarkov on August 3, 2003 12:47 AM

an excerpt from "Yes, Prime Minister":


". . .I had seen the party opinion poll as an insuperable obstacle to changing the Prime Minister's mind. However, Humphrey's solution was simple: have another opinion poll done, one that would show that the voters were against bringing back the National Service [conscription]. I was somewhat naive in those days. I did not understand how the voters could be both for it and against it. Dear old Humphrey showed me how it's done.

The secret is that when the Man In The Street is approached by a nice attractive young lady with a clipboard he is asked a series of questions. Naturally the Man In The Street wants to make a good impression and doesn't want to make a fool of himself. So the market researcher asks questions designed to elicit consistent answers. Humphrey demonstrated the system on me

"Mr. Woolley, are you worried about the rise in crime among teenagers?"
"Yes," I said.
"Do you think there is a lack of discipline and vigorous training in our Comprehensive Schools?"
"Yes."

"Do you think young people welcome some structure and leadership in their lives?"
"Yes"

"Do they respond to a challenge?"
"Yes"

"Might you be in favor in reintroducing National Service?"
"Yes."

Well, naturally I said yes. One could hardly have said anything else without looking inconsistent. Then what happens is that the Opinion Poll publishes only the last question and answer. Of course the reputable polls don't conduct themselves like that. But there weren't too many of those. Humphrey suggested we commission a new survey, not for the Party but for the Ministry of Defense. We did so. He invented the questions there and then:

"Mr. Woolley, are you worried about the danger of war?"
"Yes," I said, quite honestly.

"Are you unhappy about the growth of armaments?"
"Yes."

"Do you think there's a danger in giving young people guns and teaching them how to kill?"
"Yes."

"Do you think it's wrong to force people to take up arms against their will?"
"Yes."

"Would you oppose the reintroduction of National Service?"
I'd said "Yes" before I'd even realized it, d'you see? Humphrey was crowing with delight. "You see, Bernard," he said to me, "you're the perfect Balanced Sample". . ."

Posted by: roublen vesseau on August 3, 2003 01:11 AM

Just watch the copiuos Fox slanting...of course they are biased. Does anyone actually believe the "fair and balanced" schtick? Expecting real numbers instead of propaganda from them is naive.

Posted by: ntb on August 3, 2003 02:56 AM

My very unscientific add to this issue:
Fox asks your opinion of "politicians"; Zogby asks your opinion of "people."

Knowing that I am rating a list of politicians I may tend to rate, for example, Bush, higher than if I am expecting a list that may contain people I actually admire.

For example, "I'm going to read you a list of death row inmates; tell me for each...". Now I could just say unfavorable for each but then the poll loses it point, now doesn't it? So, in my mind I am ranking each proposed inmate against my mental list of all such and ranking them from best to worst (ok, least bad to most bad). This may not be the "correct" response to the question actually asked, but may be an effect of comparing a politician to "politicians" versus comparing a politician to other "people."

Posted by: dunno on August 3, 2003 04:48 AM

Simple - who is on your phone list - note that almost all polls have consistent places in their bracket - Harris and ARG at the low end, Fox at the high end.

Who is on your phone calling list is a relatively simple way of making a survey more or less conservative, and there are some relatively simple ways of skewing a phone sample - by calling at particular times of the day.

Posted by: Stirling Newberry on August 3, 2003 06:20 AM

Never would it occur to me NOT to appear conservative in a survey by Fox. Never would I trust a Fox survey to be confidential. So, I would be what they wish me to be.

Posted by: lise on August 3, 2003 07:51 AM

Oh and next your going to tell me that real estate agents don't maximize price. The middleman serves so many masters.

Posted by: Ben on August 3, 2003 09:20 AM

Zogby gets different results because it gives 4 choices and then aggregates the top 2 and bottom two for approval/dispproval.

As to the more general question of why Fox regularly gives bush his "best" numbers, I have no idea..

Posted by: Atrios on August 3, 2003 11:30 AM

it would appear there is a need for an organization or institution that evaluates the reliability or "truthfulness" of polls against some simple norms that an simple citizen might understand.

Posted by: Honza on August 3, 2003 12:04 PM

it would appear there is a need for an organization or institution that evaluates the reliability or "truthfulness" of polls against some simple norms that an simple citizen might understand.

Posted by: Honza on August 3, 2003 12:05 PM

... so why does Zogby come in under 0% once? That's just, like, weird.

Posted by: Dan on August 3, 2003 12:15 PM

I see Atrios and I are total agreement on the explanation of the Zogby Anomaly. But -- as he says -- when you look at polls in general, it really does appear that Fox's polls have a conservative bias of several percent (and that that the Harris and Pew polls have a similar liberal bias).

Posted by: Bruce Moomaw on August 3, 2003 04:28 PM

Another possible factor is the timing of when polls are conducted. (not time-of-day.) If there is high day-to-day variability in approval, perhaps depending on what stories have been in the news lately, when you choose to take your poll could have an effect.

The pro&anti-Bush favorability spread page at http://www.pollkatz.homestead.com/files/favorability.htm shows Fox the most pro-Bush with a +8.7% bias (baseline pretty arbitrary); but ABC isn't far behind with a +7.99%.

Posted by: Ali Soleimani on August 3, 2003 05:51 PM

Another possible factor is the timing of when polls are conducted. (not time-of-day.) If there is high day-to-day variability in approval, perhaps depending on what stories have been in the news lately, when you choose to take your poll could have an effect.

The pro&anti-Bush favorability spread page at http://www.pollkatz.homestead.com/files/favorability.htm shows Fox the most pro-Bush with a +8.7% bias (baseline pretty arbitrary); but ABC isn't far behind with a +7.99%.

Posted by: Ali Soleimani on August 3, 2003 05:52 PM

OK, so the abolute numbers are different, but the direction and rate of travel are remarkably consistent among all polls. This is why the politicians pay more attention to the trend of the poll numbers. Polls can be calibrated into likely votes and a good pollster can come pretty close. As long as one knows that the numbers are consistently biased, then they can be informative.

I might point out that there is probably more good information in these poll numbers than in the unemployment numbers which have been manipulated for political purposes..

Posted by: bakho on August 3, 2003 06:44 PM

As long as you're displaying this graph, let me point out an interesting visual feature: three fuzzy lines, each trending downward, the slope of each successive line being *steeper than* the previous one.

To see how odd this is, check out the pollcatz graphs that compare Bush II's Gallup approval ratings with those of past presidents:

http://www.pollkatz.homestead.com/files/gallupcharts2.gif
http://www.pollkatz.homestead.com/files/gallupcharts1.gif

Posted by: Canadian Reader on August 3, 2003 07:10 PM

Basically, without war and terrorism, Bush would be nowhere. I read somewhere that, taking away crises like 9-11-01 and the recent Gulf War II,
he loses 0.06% of his support per day (essentially
the slope of the downward trends of his support).
Hit the trifecta, indeed! Bush needs Bin Laden and Saddam more than they needed him.

Posted by: non economist on August 3, 2003 08:41 PM

Bruce,
Actually, you might note Atrios was pointing out that Zogby has a total of four categories, and lumps the top two into "approve", not just the bottom two into "disapprove". The questions are different, but I'm not sure the "lumping" process is thus sufficient to explain the descrepancy.

Pollkatz has also done a statistical analysis of the various pollsters -- Fox, far and away, consistently has the lowest "disapproval" scores of any (there are 13) major pollster.

http://www.pollkatz.homestead.com/files/favorability.htm

Jonathan

Posted by: Jonathan on August 4, 2003 07:24 AM

To Jonathan: I know that! Zogby, to repeat, lists "excellent" and "good" as "positive", and "fair" and "poor" as "negative", which is a very strange thing to do with those folks who rate a guy as "fair" -- about half of them, when asked a flat-out approve/disapprove question, will say they approve of him.

Posted by: Bruce Moomaw on August 4, 2003 07:38 AM

... so why does Zogby come in under 0% once? That's just, like, weird.

Check the title of the graph: it's the difference between "approve" and "disapprove" ratings, not absolute approval. So, under 0% means that there were more "disapprove" answers than "approve."

Posted by: YT on August 4, 2003 02:12 PM

A few years back a shocked NY Times ran the alarming headline "One in Three Americans Doubts Holocaust Occurred". All kinds of hand-wringing followed -- *how was it possible* that one-third of poll respondents could say that they doubted that Nazi's extermination of the Jews ever occurred???

The poll question was this:

"Does it seem possible or does it seem impossible to you that the Nazi extermination of the Jews never happened?"

After the stroy broke, the same pollster rephrased the question and asked it again as this:

"Does it seem possible to you that the Nazi extermination of the Jews never happened, or do you feel certain that it happened?"

The number of those giving the troubling answer dropped to 1%. The Times ran another story saying "never mind".

That's the effect of the phrasing of a question, and of the press's interpretation of the answer.

Posted by: Jim Glass on August 4, 2003 10:09 PM

The differences are on the disappproval and no opinion side. The Fox poll includes a “can’t say” option not included in the stated question, but there’s no downside to using it. Zogby gives you the narrower option “you are not familiar enough to form an opinion”, which amounts to admitting you are a gormless hick, as bad as not having heard of him. Everybody knows Fox are to the right of Genghis Khan, so ”can’t say” could well include mild disapprovers who don’t want to offend or challenge the interviewer. This theory reconciles the data more or less.
One of the polls includes the significant category “neutral”. These are the swing voters.

Posted by: James Wimberley on August 6, 2003 03:57 AM

Belatedly, I'd note that the "favorable/unfavorable" questions quoted above are NOT the same as the presidential job approval question. See http://www.pollingreport.com/BushJob.htm for the job approval wordings ... except, unfortunately, Zogby's ... when I go to zogby.com it appears the full crosstabs, presumably with question wordings, are available only if you buy them. (Survey research community standards dictate that Zogby should release the wordings upon request, presumably without charge, but I don't have time to pursue this myself at the moment.)

Generally, previous posters are correct that even slight differences in wording can have significant effects. Among other factors to consider:

-- Question order: Many pollsters ask presidential job approval as the very first substantive question (after screeners for registered or "likely" voters), the theory being that that gives you the cleanest read. For example, if you started with a battery of questions about the economy at a time and followed with pres job approval, if the economy is in the crapper you could bias the approval question downward (there have been many studies proving this kind of order effect).

-- Someone mentioned "likely" vs registered voters -- typically pollsters won't screen for likely voters until the last month or so before and election, as it's questionable whether how reliable the "likely" screens will be any farther out before an election. But close to an election, typically a likely voter pool will be a little more Republican than RVs, because Republicans tend to turn out more reliably.

-- The sample itself: Last I checked, Zogby draws samples only from listed phone numbers, which means he systematically excludes perhaps 30% of US households (up to 60% in California). Zogby asserts there's no difference between respondents at listed vs unlisted numbers and therefore it's not worth the extra expense to do an "RDD" sample (random digit dial, where a company that specializes in this sort of thing provides a list of known residential landline phone exchanges and the survey shop randomly generates the last four digits, thereby getting unlisted as well as listed numbers). Most other pollsters disagree.

-- Zogby is known for weighting to party ID, meaning he has some thoughts on what the appropriate mix should be (usually based on past exit polls) and he weights to that. In other words, if his sample comes in 30% D, 30% R and 40% independent but he thinks it should be 25 D, 35 R and 40 I, he'll weight down the D responses and weight up the Rs. This is common practice among pollsters for political campaigns but not among the public (media) pollsters. Zogby also applies various other weights besides the standard "ARSE" (age, race, sex, education), as per a Washington Post profile a couple years back, but hasn't to my knowledge been forthcoming in detailing them.

-- jonathan mentioned the possible effect of Fox sponsorship being mentioned in the intro. It is standard practice to identify the poll's sponsor in the intro of a media poll (not necessarily for a campaign poll, as that could bias the responses, although at the end of the survey the interviewer should disclose the sponsor if asked). Would be very interesting to see a test of this. My hypothesis would be that there could indeed be an effect, as respondents who believe Fox has a rightward tilt might either be more likely to respond if they're conservative or more likely to hang up if they're not.

Honza, not sure this is exactly what you're looking for (not updated too frequently at times) but it's a start: http://www.ncpp.org/presspost.htm , the Polling Review Board of the National Council on Public Polls.

Posted by: a pollstah on August 12, 2003 06:36 AM
Post a comment