To: Interested Parties
From: Terry Sullivan, Texans For Kay Campaign Manager
Date: November 3, 2009
Re: The Poll That Says Kinky Friedman Will Be The Democratic Nominee
Polls go up and polls go down but some polls, like today's internet-based University of Texas/Texas Tribune Poll, have proven to be unreliable on many occasions.
Internet polls - like the UT/Tribune poll - are historically unreliable and have been questioned by highly-regarded pollsters and media outlets in Texas and across the nation. As Rutgers University polling expert Cliff Zukin said in 2006, internet-based surveys lack "a basis in scientific validity." So much so, that major news organizations - including The Associated Press, The Washington Post and The New York Times - will NOT report on internet-based polls.
Even more troubling is that the UT/Tribune poll was conducted by YouGov/Polimetrix which has proven unreliable in the past. As was reported by The Wall Street Journal, polling conducted by the firm in 2008 had a higher median error than any phone pollster. For example, in the final week of the presidential election, YouGov/Polimetrix's polling showed that John McCain would win the state of Indiana by eight points. In fact, Barack Obama won the state by nearly one percentage point (.9%).
Moreover, in the information provided by The Texas Tribune, there are serious questions as to if this poll is representative of Texas voters:
· First, the UT/Tribune poll is not a random sample of registered voters. YouGov/Polimetrix has an 'opt-in' policy for its respondents. The survey respondents are people who have agreed to be a part of their sample pool in the past. This means respondents are not randomly selected from voters, which introduces some selection bias in the sample and could skew results. Typically, these respondents are surveyed multiple times per week on different topics and develop some survey fatigue as they pursue prizes like tote bags, gift certificates, t-shirts and even cash (http://www.pollingpoint.com/rewards/). Results are skewed toward heavy-Internet users: adults under 35 and men. And if you would like to sign up, click here: http://www.pollingpoint.com/.
· Second, the survey does not release demographic results for the Republican and Democratic primary voters. The demographic results they do release suggest a younger population than one would expect in a Republican primary election. In this survey, 27% of respondents were under the age of 34. In the 2008 Republican primary election for President, just 13% of respondents were under the age of 30. In the 2008 Democratic Primary election, just 16% of respondents were under the age of 30. Not only is this sample too young to reflect Republican primary voters, it's too young to reflect primary voters overall.
· Third, the poll's use of matching and weighting manipulates the sample and can lead to inaccurate results. Over 1,000 respondents were interviewed (1,152), but the survey was matched down to 800 respondents. Applying such large weights and disregarding survey responses greatly calls into question the accuracy of the final numbers.
· Fourth, it is highly troubling that the UT/Tribune poll relies on a sample of just 357 Republican primary voters who have pre-registered to participate in YouGov/Polimetrix's internet-based survey. Most traditional polls use samples that are double that number with a margin of error significantly less than today's poll. As The Dallas Morning News' Wayne Slater wrote this morning, the UT/Tribune poll's margin of error is "nearly twice as large as conventional polling standards for a horserace question."
And the track record of internet-based polls in Texas is highly questionable. In the 2006 Texas governor's race, campaigns and journalists alike widely questioned the accuracy of the Zogby poll - after it produced questionable results.
At that time, Texas Monthly's Paul Burka wrote that journalists should especially dismiss the internet-based Zogby poll because "it is not a random sample." That same reasoning would apply to the UT/Tribune poll. And once again, Paul Burka is raising alarm over this internet-based poll calling its methodology "of questionable value."
Most respected pollsters and journalists would conclude that the only way to obtain a truly representative poll is through live calling. That has yet to be done in Texas in this election save for our campaign's internal polling of Republican primary voters which shows a completely different race and Kay Bailey Hutchison in a strong position.
And finally, does anyone really think that Kinky Friendman is primed to win the Democratic primary for Governor?
Important Background Information...
Texas Reporters Are Beginning To Raise Questions About The University Of Texas/Texas Tribune Internet Poll:
The Dallas Morning News' Wayne Slater Said The Texas Tribune Poll "Has Some Problems," Including The Poll's Opt-In Mechanism And Large Margin Of Error. "But the poll has some problems. The survey was of 800 registered voters and conducted on the Internet. Participants self-register to be in the survey, a departure from traditional polling. For the Hutchison/Perry matchup, the survey size was 357 self-described likely Republican voters with a margin of error is 5.19 percentage points, nearly twice as large as conventional polling standards for a horserace question." (Wayne Slater, "Perry Leads Hutchison In New Poll, But Is It Accurate?" The Dallas Morning News' "Trail Blazers" Blog, trailblazersblog.dallasnews.com, 11/3/09)
Texas Monthly's Paul Burka Said He Was "Skeptical" Of The Texas Tribune Poll, Calling The Poll's Methodology "Of Questionable Value." "I want to say a couple of things about the UT/Tribune poll. The first is that we are lucky to have it. Polls cost money, and without this poll, we would have to rely on the hard-pressed daily newspapers to know where the race stands, or Rasmussen's occasional reports on the governor's race. The second is that I continue to be skeptical about the poll's methodology. The gold standard for polling primary elections is to acquire the list of people who actually voted in a party primary in previous elections and limit the sample to these names. In this way you have a 'pure' sample of likely primary voters. Anything short of this, such as asking voters whether they intend to vote in a primary election, is of questionable value." (Paul Burka, "UT/Texas Tribune Poll: Perry +12; Medina at 7%," Texas Monthly's "Burka Blog," www.texasmonthly.com, 11/3/09)
The UT/Texas Tribune Internet Poll Was Conducted By YouGov/Polimetrix, Which Has A Questionable Track Record:
The October University Of Texas/Texas Tribune Survey Was An Internet Poll Conducted By YouGov/Polimetrix. "The October 2009 Texas Statewide Study was designed by researchers in the UT- Austin Department of Government and conducted by YouGov/Polimetrix, a firm with demonstrated success in internet polling. YouGov/Polimetrix accomplishes internet polling through a unique sampling procedure know as 'matched random sampling.' The firm begins with two lists: (1) a list of all adult 'consumers' in Texas (covering approximately 95 percent of the adult population), and (2) a list of people who have agreed to take YouGov/Polimetrix's surveys." ("Sampling And Weighting Methodology For The October 2009 Texas Statewide Survey," Texas Tribune, static.texastribune.org, Accessed 11/3/09)
· YouGov/Polimetrix's Internet Polling In The 2008 Presidential Election Had A Higher Median Error Than Any Phone Pollster, "Including, Notably, An Eight-Point Margin For Sen. McCain In Indiana." "If so, Internet polling has a ways to go by 2012. The only pollster active in every state in the final week, YouGov/Polimetrix, surveyed people from among a panel who were randomly selected and invited to participate in its political polls. In many states, the sample size was under 500. And YouGov's median error of 4.1 percentage points -- including, notably, an eight-point margin for Sen. McCain in Indiana -- was higher than for each of the active phone pollsters." (Carl Bialik, "Polls Foresaw Future, Which Looks Tough For Polling," Wall Street Journal, 11/6/08)
A Recent Study By Stanford University Researchers Called Into Question The Polling Methodology Used By YouGov/Polimetrix:
A New Study By Stanford University Researchers Found That "Non-Probability Internet Surveys Were Less Accurate, And Customary Adjustments Did Not Uniformly Improve Them." "In the most extensive such analysis to date, [Stanford University's] David Yeager and Prof. Jon Krosnick compared seven non-random internet surveys with two surveys based instead on random or so-called probability samples. The non-probability internet surveys were less accurate, and customary adjustments did not uniformly improve them. While the random-sample surveys were 'consistently highly accurate,' the internet surveys based on self-selected or 'opt-in' panels 'were always less accurate, on average, than probability sample surveys, and were less consistent in their level of accuracy,' the researchers said. Further, they said, adjusting these samples to known population values had no effect on accuracy (and in one case even worsened it) as often as that process, known as weighting, improved it." (Gary Langer, "Study Finds Trouble For Opt-In Internet Surveys," ABC News' "The Numbers" Blog, blogs.abcnews.com, 9/1/09)
Major News Organizations Will Not Report On Internet-Based Polls:
The Policy At ABC News, The Associated Press, The New York Times, And The Washington Post Is Not To Report On Internet-Based Polls. ABC News' Gary Langer: "The results shouldn't be a surprise; as I've reported previously ... non-probability samples lack the theoretical underpinning on which valid and reliable survey research is based; our policy at ABC News, as at several other national news organizations (including The Associated Press, The New York Times and The Washington Post) is not to report them." (Gary Langer, "Study Finds Trouble For Opt-In Internet Surveys," ABC News' "The Numbers" Blog, blogs.abcnews.com, 9/1/09)
Rutgers University Polling Expert Cliff Zukin Cautioned Against Internet-Based Polls Saying They Lack "A Basis In Scientific Validity." "When reached by phone last week, Cliff Zukin, a political science professor and polling expert at Rutgers University, suggests that journalists should generally be wary of any Zogby interactive poll. 'The Zogby stuff, on scientific grounds, is quite questionable,' says Zukin. 'Online, Internet, opt-in polling, where people volunteer to be respondents, doesn't really have a basis in scientific validity. There are two kinds of samples in the world. There are probability samples, and there are non-probability samples.' The Zogby interactive polls, says Zukin, clearly fall into the latter camp. ... Another problem with Internet-based polling, says Zukin, is that, in general, Web and email-based surveys tend to overvalue the opinions of young people. A group that is notoriously lousy at showing up to actually vote." (Felix Gillette, "How Reliable Is The Zogby-Journal Poll?" Columbia Journalism Review, www.cjr.org, 9/18/06)
In 2006, Internet-Based Pollings In Texas Was Widely Questioned:
In 2006, The Perry Campaign QuestionedThe Reliability Of Internet Polls When A Zogby Poll Put Them At 31 Percent. ""[T]he Zogby poll had [Perry's reelection support] at nearly 31 percent. ... Afterwards, Perry aides questioned the reliability of the polls -- one of which, the Zogby interactive poll, relies upon an untraditional methodology that has raised some eyebrows in past elections. According to its Web site, Zogby interactive polls tap a self-selected group of respondents who sign up for surveys online and then respond to specific questions via email. So was the Perry camp correct to question the Internet-based poll?" (Felix Gillette, "How Reliable Is The Zogby-Journal Poll?" Columbia Journalism Review, www.cjr.org, 9/18/06)
Texas Monthly's Paul Burka Discredited The Zogby Poll's Online Polling Methodology, Saying "It's Not A Random Sample" And Does Not "Screen For Likely Voters." "'The [Zogby-Journal] poll's Web site describes the process as 'interactive' -- that is, it's an Internet poll, based on a database of individuals who have signed up to participate,' writes Burka. 'It is not a random sample; the polling organization solicits responses by e-mail. In addition, the poll takers make about 20 to 50 phone calls in the state where a race is taking place. The poll does not mention a screen for likely voters.' 'As I have written repeatedly, the poll that I think is the least credible is Zogby/Wall Street Journal/Battleground States,' adds Burka. 'I can't believe the Journal allows its name to be attached to this so-called poll.'" (Felix Gillette, "How Reliable Is The Zogby-Journal Poll?" Columbia Journalism Review, www.cjr.org, 9/18/06)
Not surprisingly, aides to two gubernatorial hopefuls who didn't finish first among their party brethren in the inaugural UT/Texas Tribune poll are questioning the poll's methodology. Vince Leibowitz, campaign manager for Democrat Hank Gilbert, has been insisting to everyone and anyone today that surveys taken over the Internet are worthless measures of public opinion. "People may complain about the reliability of telephone polls, but the reliability of this poll is so far from being beyond reproach I don't even know what else to say," he told Wayne Slater of the Dallas Morning News. Meanwhile, Terry Sullivan, campaign manager for Republican Kay Bailey Hutchison, distributed a long memo questioning whether our "unreliable" poll (there's that word again) is "representative of Texas voters." "Most respected pollsters and journalists would conclude that the only way to obtain a truly representative poll is through live calling," Sullivan wrote.
Mr. Leibowitz and Mr. Sullivan are surely entitled to their opinions, and we're happy to see them ping-pong across the dreaded Internet; just spell our name right, fellas. And our pollsters, Jim Henson and Daron Shaw, are more than capable of explaining and defending themselves. But it's worth noting the obivous: that in polling, as in life, where you stand is where you sit. It has been ever thus that when a poll produces a finding favorable to your candidate, the methodology is perfect and the pollsters are geniuses; as soon as the very same poll produces an unfavorable finding, the same methodology is fatally flawed and the pollsters are idiots. Does anyone doubt that if Hutchison were ahead of Perry in the poll we released this morning, her campaign would be crowing about the brilliance and insight of our data collection? Or that Gilbert would be doing the same? Well, one person doubts the latter: Vince Leibowitz, who maintained in a phone call to me early today -- I assume he had a straight face -- that even if Gilbert had come out in front of the pack, he wouldn't have sent out a press release touting that result, so terrible is our approach to polling. Sure, Vince. You would have passed up the chance to tell the world that your candidate is in the lead.
Honestly, I don't blame Leibowitz for making such a fuss about this, and I don't blame Sullivan either. They're just doing their jobs. As are our pollsters. As are we all.
Terry Sullivan comes off as "FREAKING OUT" and "OVERREACTING" on this one.
ReplyDelete