Tuning the Truth-O-Meter

Making Truth-O-Meter rulings is the hardest thing PolitiFact does. We follow a set of principles to keep our ratings consistent and we have a process that involves at least four journalists in every rating.

Most of the time, we make the right call. Occasionally, we don't.

Such is the case today with a rating on a claim by President Barack Obama that, "In the last 22 months, businesses have created more than 3 million jobs. Last year, they created the most jobs since 2005." After Obama’s State of the Union speech, we rated the claim Half True. But this morning we reviewed our work and decided to change it to Mostly True because we concluded Obama was not crediting his policies as directly as we originally thought.

The process that got us to Half True illustrates the challenge we face in rating the many claims of blame and credit in our political discourse.

In our first couple of years, we treated many of those claims very literally. If someone said jobs had gone up since a governor was in office, and we found the numbers backed it up, the statement earned a True.

About a year ago, we realized we were ducking the underlying point of blame or credit, which was the crucial message. So we began rating those types of claims as compound statements. We not only checked whether the numbers were accurate, we checked whether economists believed an office holder's policies were much of a factor in the increase or decrease.

We give a lot of Half True ratings because the numbers are often right, but experts repeatedly tell us that the policies of a single executive have a relatively small impact in a big and complex economy.

In approaching Obama's claim on jobs in the State of the Union address, we applied that principle. But after hearing lots of feedback from readers, we've concluded we gave too much weight to the linkage. And we've changed the rating to Mostly True.

We started checking this one about 10 hours before the State of the Union address after PolitiFact Texas Editor Gardner Selby heard the claim from Obama adviser David Plouffe in a TV interview. Gardner suggested we check it.

Realizing that Plouffe's line would probably be in the State of the Union, PolitiFact National writer Lou Jacobson, who does many of our fact-checks on economic claims, began looking at the numbers. By early afternoon, Lou had his reporting done and had concluded that the numbers were right.

At that point, Lou, deputy editor Martha Hamilton and I had several conversations about the rating. We wrestled with whether it deserved a Half True or a Mostly True and could not reach a conclusion. We decided that it would depend on how directly Obama linked the jobs numbers to his policies.

(A quick explanation of our process: A reporter does the research, writes the fact-check article and recommends a Truth-O-Meter rating. After the story is edited, it goes to a panel of three editors that decides on the final ruling.)

When we got the advance text of Obama's speech about 20 minutes before he began, we saw how Obama was going to use the line. The passage:

". . . and we lost another four million before our policies were in full effect. Those are the facts. But so are these. In the last 22 months, businesses have created more than three million jobs. Last year, they created the most jobs since 2005. "

Lou, Martha and I had another conversation about the rating and whether it should be Half or Mostly True. At various points, each of us switched between Half and Mostly True. Each of us felt it was right on the line between the two ratings (unfortunately, we do not have a rating for 5/8ths True!).

We brought another editor, deputy government & politics editor Aaron Sharockman, into the conversation and he too was on the fence. Finally, we decided on Half True because we thought Obama was implicitly crediting his own policies for the gains.

I mention that back and forth to show you how thoroughly we discuss our ratings and how much thought we put into them. We know our readers won't agree with every call we make, but we hope you'll appreciate the thoroughness of our process.