Portland mayor says nearly 87 percent of food scrap pilot participants were satisfied with the program

Fresh off winning a resolution to ban plastic bags in Portland grocery stores, Mayor Sam Adams is pitching another way to make the city ever greener: curbside composting.

The mayor wants to rework the way the city collects its residents’ garbage. Under his proposal, Portlanders would get composting pails for their kitchens which they would empty into their yard debris roll carts. The debris and compost would be picked up every week while regular garbage pick-ups would drop to every other week.

About 2,000 households have been doing this since May 2010 as part of a pilot program. The City Council is set to vote on the proposal in the next week or so with the goal of getting the whole city composting by the end of October.

In selling this new idea to residents, the mayor and his staff have been citing all sorts of statistics, including this pretty persuasive one: "Of the 2,000 Portland households in the year-long pilot, 87 percent of participants reported being satisfied with the overall system."

It didn’t take long before the stat found its way around the media. Not only does it appear at least twice on city websites, but it also made its way into reports over at OPB News and KGW News (8).  The statistic also caught the eye of local blogger Victoria Taft, who questioned its veracity.

We wanted in on the action, so we called the mayor’s office to check on the poll. An hour later, we had a copy in our inbox.

A quick look at the first page of the poll seemed to bear out the mayor’s claim -- at least partially. According to the survey his office conducted, 63 percent were "very satisfied" with the program, while another 24 percent were "somewhat satisfied." Add those together and you have that 87 percent satisfaction rate. (The others were either "not satisfied" or "don’t know.")

Those numbers are right. But we had two issues with the survey: the design and the sample.

The design of the poll seemed to tilt the response toward the positive end. You only have three categories "very satisfied," "somewhat satisfied" and "not satisfied" and the mayor’s office lumps the first two together to mean "satisfied."

Typically, however, there’s a bracket, said Stuart Elway, a pollster out of Seattle.

"You can get into arguments about how many points there should be in a scale," Elway said. "Typically you would have a balanced number, so you'd have two positive and two negative."

We know for certain from the mayor’s survey that 63 percent of the respondents were definitely satisfied. That second category, though, is squishy. Would those folks, if given the option, have cited mild dissatisfaction? No one can be certain.

On to the sample.

According to the report the mayor’s office sent us, the survey only covered 391 responses. That’s a difference from the claim in our ruling statement. The statement makes it sound as though all 2,000 pilot homes responded and that, of them, 87 percent liked the program.

We asked the mayor’s office about this discrepancy. Dan Anderson, the mayor’s spokesman, told us that we were right -- only 20 percent responded. But, he said, the statements on the city’s website were still accurate. That 20 percent, he said, is statistically significant and could, mathematically speaking, represent the whole.

"This isn’t Survey Monkey," he said referring to a website that allows users to make quick and dirty polls. "It’s sophisticated, it’s accurate, it’s demographically diverse. We’re confident that it’s representative of what those in the pilot program felt of the program."

The website, he said, could be clearer, but was still accurate. He promised to update it. He also sent us some minutes from the August 10, 2011 City Council meeting in which Adams speaks about the survey.

"The 20 percent in terms of return rate in a neighborhood of 2,000, we do -- we do polling, our recent polling by the city of Portland, actually polled a much smaller sample," Adams said. "So 20 percent is representative. We think, adequately representative in terms of statistical validity."

But he’s not entirely right.

Indeed, a 20 percent response rate to a mail survey is on the high end, said Elway. Still, when the respondents are self-selected, as they were here, it’s hard to tell whether they’re truly representative.

"You really can't apply the laws of probability for a mail survey," Elway said. "But what you can do is say ‘Of the 20 percent that we got back, how well do they match the population?’ The more of those kinds of measures you have, the more confidence you can have that the results are valid.

We asked the mayor’s office if they’d taken a look at whether the respondents reflected the larger group’s demographics, but they hadn’t collected many of the details.

The mayor’s office made one other point during our discussion: Usually self-selecting respondents trend negative. If you really dislike something, you’re motivated to let people know.

Elway said that is sometimes true, but that it’s probably not as much of an issue for a pilot group.

If anything, he said, it’s possible the poll might be skewed positive.

"It doesn’t sound like a totally bogus poll or anything," he said. But "if I were the client looking at this, I would look at the positive number I got back with squinty eyes.

So where does all this leave us? Well, we’re not sold that the satisfaction rate is as high as 87 percent, given the survey design and the way the sample was collected. But, it does seem obvious that, among the pilot homes, the program is relatively popular.

We’ll give this statement a Half True.

Featured Fact-check

Scott Bruun

stated on September 17, 2010 in a statement on his website

"The size of our federal government has more than doubled since the year 2000."