When fact-checkers disagree

Editor’s note: When we first examined this statement from Vice President Joe Biden, we rated his statement Mostly True. We noted in that report, published Oct. 20, 2011, that the numbers he gave for rapes seemed questionable. Shortly afterwards, we explained why our original report was at odds with other independent fact-checkers in this story. Since then, we found more definitive evidence that Biden's numbers on rape are wrong. We have re-reported this item and given it a new rating of Half True. We have also archived our original report with the Mostly True rating. You can also read the reader reaction we received to this story.

Joe Biden has been making comments recently about connections between murders, rapes, cops and budget cuts. Factcheck.org called his words a "whopper." The Washington Post’s Factchecker said, "absurd." And PolitiFact? We said "Mostly True."

It was a disconcerting schism for us. It’s much more satisfying when the major fact-checking organizations agree, because we know we’ve reached the same conclusions independently.

But after reading the three reports closely, we began to understand how we arrived at different conclusions, even though there were no obvious errors of fact-checking to correct.

Here, we want to highlight a few of the differences between our reports and ask what you think.

In brief, the argument Biden made is that cuts to police forces can result in increased murders and rapes. His point is that President Barack Obama’s jobs bill could help fill state and local budgets, which could in turn make cities safer.

For his example, Biden chose Flint, Mich. The statement we chose to check was one Biden made on Oct. 12, when he said,  "In 2008, when Flint had 265 sworn officers on their police force, there were 35 murders and 91 rapes in this city. In 2010, when Flint had only 144 police officers, the murder rate climbed to 65, and rapes … climbed to 229." We looked at that statement and ruled it Mostly True.

We encourage you to read our full report, but we concluded that he was largely correct about increasing murders and a decreasing police force. Flint had to cut 66 officers in 2010; that same year, its murder rate reached a historic high. The numbers Biden gave for murders were slightly different from the FBI numbers, but they weren’t much different. An expert told us local police often keep more up-to-date numbers than the FBI, which could explain the difference.

We knew the rape numbers Biden gave were at odds with FBI numbers, but we couldn’t fully explain why. We accounted for that in our ruling by giving him a Mostly True. In retrospect, perhaps we should have given him a Half True for the discrepancies on the rape statistics alone.

Our friends at Factcheck.org, meanwhile, checked two statements from Biden, including one Biden made on Oct. 18, almost a week after the comments we were checking. In those Oct. 18 comments, Biden amped up his claim, stating that the number of rapes in Flint had "quadrupled" and the number of murders had "tripled." (Read Factcheck.org’s report.) That is clearly wrong, even if you’re just using the numbers Biden cited the week before. Additionally, Factcheck.org focused heavily on the rape numbers, which as we had noted were the most questionable part of the claim. The city of Flint has acknowledged problems with the numbers and said it intends to release fuller information, but has not yet done so.

The Washington Post's Factchecker noted the discrepancies in the numbers compared with the FBI statistics, but also concentrated on whether there was a link between police force and increased crime.

So is there a connection between how many police officers there are in a city and increases in crime? Experts quoted by the Factchecker and by PolitiFact said it’s one factor among many. The Factchecker counted this as a strike against Biden’s claim, concluding, "Even if one believes there is a link between crime and the number of police—which is debatable and subject to many caveats—there is no excuse to make the dramatic claim that more people will die or be raped without additional funds for police." Here at PolitiFact, we counted that as supporting evidence for Biden’s claim that crime was going up in Flint while the police force was being reduced.

All of this leads us to a few conclusions.

• It matters what specific statement fact-checkers choose to check. PolitiFact looked at one of Biden’s earlier statements. As Biden continued to make the point, he expressed it with more vehemence and exaggerated it, leading to the subsequent wrong claim that Factcheck.org looked at.

• Causal claims about complex social phenomena remain a fact-checking challenge. All the experts said police force size was one factor among many when thinking about increases in violent crime. Should that weigh in favor of a statement like Biden’s or against it? It’s not clear cut.

• Independent fact-checkers often agree -- but not always. It doesn’t happen often, but sometimes the major fact-checking organizations are not going to reach the same conclusions. Often, we agree on the underlying facts, but we weigh the evidence differently or see the context in different ways. Sometimes we make judgments that we might wish we could go back to and tweak slightly. Particularly on our Truth-O-Meter, we find legitimate differences over whether how certain statements should be rated:  "Mostly True" vs."Half True," or "Mostly False" vs. "False," and even "Mostly True" vs. "True."

This is the part where we’d like to hear from you. We want to take some time to consider re-rating our Biden fact-check or if more reporting is required. Send us your thoughts -- you can e-mail [email protected] -- as we consider the evidence.