Wis. Gov. Scott Walker says the St. Norbert College poll has been repeatedly wrong

As Wisconsin Democrats geared up to begin the process of recalling Gov. Scott Walker, a public opinion survey released Nov. 15, 2011 contained some troubling results for the first-term Republican.

The poll, conducted by St. Norbert College in De Pere and Wisconsin Public Radio, said 58 percent of respondents believe Walker should be recalled. That was an increase from 47 percent  in an April poll conducted by the same partnership.

When asked in a television interview about the results, Walker turned the tables and focused on the poll itself:

"Literally, it's been wrong in almost every election," Walker said. "It's been completely off the mark. It's nowhere near any other poll that's been out there. I don't take any credence in that."

The St. Norbert poll has been around since 1984 and its results receive wide coverage in the state media. This is the first instance we could find of someone publicly claiming it’s fundamentally a "bad" poll.

When we asked Walker’s office to provide evidence of the governor’s claim, they sent us to state Republican Party spokesman Ben Sparks.

He made two points:

Polling sample: The result of the recent poll "is essentially worthless" because pollsters interviewed 426 adult residents and did not narrow the group to likely voters, he said. That reduces its accuracy, he said, because there’s no way of knowing whether or not these people participate in elections.

Track record:  In October 2010, the St. Norbert/WPR poll looked at the upcoming election for governor, pitting Walker against Democrat Tom Barrett. "That poll did not reflect the election results within the margin of error," Sparks said. He offered no additional evidence about the poll’s accuracy in previous elections.

We’ll start with a caveat: We’re examining Walker’s statement, not attempting to rate the St. Norbert poll against any others. It’s worth noting that since the St. Norbert poll  generally uses a smaller sample than most, it will have a larger margin of error.

On the question of who was polled, a separate poll conducted a few weeks earlier than St. Norbert’s poll shows how results can differ depending on the group targeted.

Public Policy Polling reported Oct. 26, 2011 that state voters were evenly divided about recalling Walker. That poll had a far larger sample -- 1,170 -- than the St. Norbert poll with a margin of error of plus or minus 2.9 percentage points. Another key difference is that Public Policy Polling interviewed likely voters; St. Norbert’s sample, as mentioned, came from all state adults.

Sparks argued that made the poll "worthless."

But his criticism is more an assessment of the poll’s predictive value, not whether it is accurate in terms of what it set out to measure. And it should be noted the pollsters did not hide who was sampled, or present its findings as representing likely voters.

On the question of the poll’s track record, we turned to University of Wisconsin-Madison political science professor Charles Franklin, founder of a website called Pollster.com. That site compiles and keeps track of all Wisconsin polls that are publicly available as well as national ones.

Franklin said one measure of the accuracy of the St. Norbert poll was to look at the results of other polls conducted at about the same time.

After all, polls are a snapshot -- not a prediction -- and influenced by advertising, debates and other developments in a campaign. So it makes sense to stick with the same time frame.

Franklin looked at two polls conducted at the same time St. Norbert researchers were in the field. For the pre-election poll, St. Norbert researchers sought out likely voters (not all adults), as did the other two polls he reviewed.

St. Norbert pollsters interviewed voters Oct. 12 to 15, 2010. Opinion Research Corp., working  for CNN/Time, was in the field Oct. 8 to 12, and the Mellman Group, a firm that works for Democrats, conducted its interviews Oct. 18 to 20.

Here are the results:

• St. Norbert: Walker 50, Barrett 41, Undecided 6, Other 2. The poll interviewed 402 likely voters and had a margin of error of plus or minus 5 percent.
• Opinion Research: Walker 52, Barrett 44, Undecided 2, Other 2. The poll interviewed 931 likely voters with a margin of error of plus or minus 3.5 percent.
• Mellman Group: Walker 47, Barrett 45, Undecided 7. The poll interviewed 600 likely voters with a margin of error of plus or minus 4 percent.

The actual results of the Nov. 2 election: Walker 52.2 percent; Barrett 46.5 percent.

So how did the St. Norbert poll do?

Franklin said there are two ways to measure this, beyond the obvious fact that it got the winner right.

The difference between the poll result and the actual outcome. St. Norbert had Walker leading Barrett 50 to 41. Walker actually got 52.2 and Barrett got 46.5. That’s .5 of a percentage point outside the poll’s 5 percentage point margin of error.

The margin of victory. St. Norbert had Walker ahead by 9 and he won by 5.7 So it was off by 3.3 points. At the very worst they underestimated Barrett by 5.5 and Walker by 2.2, Franklin said.

Franklin also looked at the results from the same St. Norbert poll for the U.S. Senate race between Republican Ron Johnson and Democratic incumbent Russ Feingold. The poll showed Johnson at 49 percent and Feingold at 47 percent.

Johnson got 51.9 percent to Feingold's 47 percent, so a 2.9 error on the margin.

Franklin’s conclusion: "At least in 2010, St. Norbert was right on both winners while over estimating the size of Walker's win and under estimating the size of Johnson's win by 3.3 and 2.9 points respectively. Those errors are both within the theoretical margin of error for the sample size they used."

We asked St. Norbert Poll director David Wegge about the most recent poll.

Why didn’t they interview likely voters?

Wegge said the most recent poll surveyed 482 adults and was an effort to measure public opinion on a variety of topics, including general public policy matters and the effort to recall Walker. He said the pollsters were not trying to assess the outcome of an election, so they didn’t limit their sample to likely voters.

Wegge, who founded the St. Norbert Survey Center (now called the Strategic Research Institute), said the organization does professional research for a variety of clients including the United Way, the Green Bay Packers, Affinity Health Systems and the Oneida Tribe.

He noted the poll fared well in a 2010 blog by Nate Silver of the New York Times, who looked at the performance of 265 pre-election polls nationally. St. Norbert ranked 45th on that list, six places behind the far-better-known CBS/New York Times and in between polls run by the Kansas City Star and the Miami Herald/St. Petersburg Times.

Finally, we looked at the results of St. Norbert’s pre-election poll from a couple of other state-wide elections. Remember, Walker claimed the poll was "wrong in almost every election."

Wegge said the same size and margin of error has been the same for years -- about 400 people and plus or minus 5 points. The pre-election polls always look at likely voters, he said.

For the pre-election poll in the 2004 presidential race: St. Norbert had Democrat John Kerry at 48 (actual result, 50) President George Bush 43 (49) and undecided at 5 (none in the actual vote). The undecideds broke to Bush, and the poll was "right."

For the 2002 gubernatorial election: Democrat Jim Doyle 41 (45) Republican Scott McCallum 33 (42), Independent Ed Thompson 6 (11) and undecided 17 (none.) That large group of undecided voters moved primarily to the losing candidates. Because of the undecideds, the poll was too low on McCallum -- outside the margin of error -- but it did get the winner and losers correct.

So what result do we get?

Walker dismissed a St. Norbert/WPR poll that showed 58 percent of state residents favored a recall as being consistently wrong. As evidence, the GOP criticized the pollsters for not quizzing likely voters. And they said the poll got the 2010 election wrong, but offered no evidence that the poll has been "consistently wrong."

In it’s most recent poll, St. Norbert wasn’t trying to predict an election outcome. It was  looking at whether the general public favored a recall. Secondly, GOP spokesman Sparks and Walker were wrong about the poll when it came to the 2010 results. The poll got that race right. And there’s no evidence that the poll has been consistently wrong in the past.

We rate Walker’s claim False.

Featured Fact-check

Eric Hovde

stated on October 3, 2024 in X, formerly Twitter

"FEMA is out of money and doesn't have money to transfer to those people affected by the hurricane ... they used the money to assist illegal immigrants."