Books: 'Policy and Evidence in a Partisan Age: The Great Disconnect'

Editor's note: We don't often interview authors or provide coverage of books, but we thought this one might interest our readers. Let us know what you think of this special feature: Send us an e-mail at [email protected].

We love discussions of the nuts and bolts of public policy, so Paul Gary Wyckoff's new book Policy and Evidence in a Partisan Age: The Great Disconnect easily made it to the top our reading list. Wyckoff believes that actual data doesn't get enough attention in policy debates, usually because people either don't understand or aren't interested in statistical evidence. We rely on anecdotes, theories and case studies to make policy decisions, even when better data exists.

This inattention to real-world evidence makes us believe that government is more powerful that it is, Wyckoff contends. We're an "oversold society," he writes, and that plays itself out in different ways on the left and the right. Conservatives, for example, believe that the government has tremendous power to suppress economic growth through tax increases. Wyckoff says the research literature says otherwise: Given the country's current debt levels, neither tax cuts nor tax increases will have much effect on economic growth. Many liberals, on the other hand, believe that public school can produce better outcomes through more spending and smaller class sizes. Wyckoff says a literature review of the empirical data shows the effects of increased spending are easily overwhelmed by the child's family and socioeconomic status.

This might sound depressing: If policy doesn't change anything, why even try? Actually, Wyckoff says, we should view policymaking through a different prism. When it comes to tax increases, we should look at programs individually and decide if the costs are worth the gains, rather than hinging the argument on economic growth. When it comes to education and welfare, we should understand that policies will effect little measureable change and ask ourselves what are society's moral and ethical obligations to the poor.

Wyckoff is a professor of government and directs the public policy program at Hamilton College in Clinton, N.Y. He teaches courses in applied statistics, ethics and cost-benefit analysis, and examines issues such as health care, education, social security and welfare.

We're not endorsing Wyckoff's views here, but his book certainly made us look at our fact-checking efforts with fresh eyes. Below is a lightly edited question-and-answer session we conducted with Wyckoff via e-mail.

You write in your book that society is "oversold" on the power of government, for both good and ill, that both liberals and conservatives believe that the government affects outcomes more than it does. To some people, that must seem pretty counterintuitive. It is the federal government, after all. What counterbalances government power?

Human decisions include a lot of factors, and what we've failed to notice is that the governmental effects are often very small in comparison to the other factors involved.  For example, we've taken steps to insure that women on welfare don't have more babies, but the childbirth decision is very complex, and the small financial penalties involved are dwarfed by other factors such as cultural values, career aspirations (or lack of them), relationships with the baby's father, the way the mother was raised, and so on. 

We try to change the location decisions of firms, but the small tax effects involved are often less important than things like the geographic preferences of the entrepreneurs, the cost of labor and energy, access to transportation, international exchange rates, and the geographic location of demand for the firm's products.

The key point is that we've oversold the behavioral effects of government policies.  Government policies can be categorized into five types: redistributional, investment, service, regulatory, and behavioral, although programs often span more than one category. What we've oversold is the power of government to change individuals' behavior. There are many effective redistributional, investment, service, and regulatory activities in which the government is very powerful.

One of the chapters in your book asks the question, "Can we spend or tax cut our way to prosperity?" You conclude the answer is no, that past data on the economy suggests that public debt counterbalances both tax cuts and spending. You say the real choices are more modest and straightforward, more like those facing ordinary families. What do you think is a current example of this dynamic?

Here's an example from The Washington Post . Recently, Gov. Kaine of Virginia spoke on the state's economic and revenue forecast.  In part, he said:

"We have kept the Virginia business climate strong. As you all know, Virginia has been named the most business friendly state in America seven times during my tenure as governor. That's why we have seen so many globally innovative companies like IKEA, Volkswagen of America, SRI, AREVA/Northrop Grumman, Canon, MeadWestvaco, Orbital, Hilton, CSC, EDS, CGI, and Rolls Royce choose Virginia for major investments in the last three years."

The idea seems to be that if we just keep taxes on business low, we can keep companies investing in Virginia and have more money for everything — education, health care, welfare, and transportation. But the literature suggests that the effects of state taxes on economic activity are very small — so small that reducing tax rates results in lower tax revenue. This means that giving tax breaks to companies means less money for schools, hospitals, and single mothers.  So states face hard choices about what they value most — more jobs at any cost, or better education for our kids, or better hospitals for patients, or better benefits for the children of poor mothers. This is very similar to a family trying to decide if they should spend their money on swimming lessons for Jimmy, a vacation for the whole family, or a down payment on a new car.

Recently, I had a conversation with an activist who said that media fact-checking doesn't mean much to him. The mainstream media is biased, he said, therefore our fact-checking is biased. My answer to him was that here at PolitiFact we cite all our sources and hope that readers can draw their own conclusions, even when they disagree with our ultimate rulings. What do you think of this exchange, and is there another answer I could have given him?

I would start by asking the activist how he gets the information he uses to form his opinions, if not through the media. Opinions must be formed based on some sort of data, or else we lose the capacity to learn about the world.  Opinions based only on instinct or internal biases trap us into a dead-end, self-perpetuating view of reality.  To paraphrase Winston Churchill, data is the worst way to form our opinions, except for all the other ones.  

If you can get your activist to go that far with you, the next question is what kind of data to use to form his opinions. Today it is possible to bypass the media and get information directly from think tanks, blogs, and Internet databases. The important point is to be aware of the quality of the data you're getting.  As the book explains, data from large-scale experiments and literature reviews of empirical work in the field is much better than low-quality data such as anecdotes, case studies, and raw figures. The latter suffers from problems with sample size (the cases may not be representative of the population) and confounding factors (some other thing may have caused the results being observed).  

How do you think the ideas in your book are reflected in the debate on health reform?

One area that has been oversold is the benefits of prevention. The theory is that if we can just prevent disease, or catch it early in its course, we can cut the costs of treatment dramatically and save lots of money. But the literature suggests over and over that it is hard to save money through screening and prevention activities, because you have to provide these programs to large numbers of healthy people in addition to the few who actually have the disease in question.  If you test everyone for diabetes, for example, you may get 50 or 100 negative tests before you find someone with the disease.  According to Congressional Budget Office chief Douglas Elmendorf, and reported in Charles Krauthammer's Washington Post column of August 14, 2009, "Researchers who have examined the effects of preventive care generally find that the added costs of widespread use of preventive services tend to exceed the savings from averted illness." This is a good example of the dangers of proceeding strictly according to our intuition without checking the data. The idea that prevention saves money makes perfect sense, so it's easy to use it to oversell a policy, but the data tell us that the world is more complicated than our simple, intuitive models.   

Do you think we as a society have enough data to make good decisions? Do we need more data? Or, is the data out there, but we don't look at it? Or some mixture?

I do think we have the data to make good decisions.  As pointed out by Ian Ayres in his book Super Crunchers , the digitization of data and the sharply reduced costs of storage and dissemination of that data make it easier than ever for individuals to get high-quality information.  The problem is that we really don't know how to analyze all that data.  As discussed in my book, if citizens and politicians have training in economics, it is overwhelmingly theoretical in orientation, with little empirical content.  The most common postgraduate training for politicians is law school, and that provides almost no instruction in statistics. As a result, we enjoy an ever-increasing stream of information, but we lack the tools to make sense of it. We're like doctors given the very latest technical devices but denied training in how to use it. The book gives readers some help in this regard by discussing the various types of information used in policy debates and explaining why some information is better than others.

I recently read another book, The Happiness Hypothesis , by psychology professor Jonathan Haidt. One of his ideas is that we are emotional creatures: We often take positions — on politics, but other things too — and then we look for evidence to support that position. Do you agree with that idea? What can we do to look at evidence in a way that is truly open, so we're not just seeking to confirm our own preconceived ideas?

Oh yes, the "filtering" of data to find information consistent with our prior beliefs is a common thread in psychology.  In his book Judgment Misguided , Jonathan Baron calls this "my-side bias."  One of the more subtle ways in which this works is that when we are confronted with data confirming our views, we accept it without question, but when we are confronted with contradictory data, we scrutinize it thoroughly, looking for holes.  As a result, different types of data face hurdles of different heights, distorting our view of the world.  

To combat this problem, Baron prescribes "active open-mindedness," by which he means actively seeking information that contradicts your views.  If you're a liberal, that means watching Fox News sometimes, and maybe reading the National Review and the Washington Times .  If you're conservative, perhaps you could check out MSNBC and The Nation , along with The New York Times .  Look at the quality of the information provided in your "normal" news sources and these alternative sources over various issues.  On each issue, which source uses high-quality, controlled studies, literature reviews, and experiments, and which uses simple anecdotes?