Fact-checking the death toll estimates from Hurricane Maria in Puerto Rico

Many news reports latched on to the number 4,645 in a new Harvard University study about the death toll in Puerto Rico from Hurricane Maria — more than 70 times above the government’s official count.

While the number grabbed headlines, it requires explanation about the methodology and what the study actually found. The number is an estimate, not a specific count of documented deaths.

Death counts after a disaster are important, because they fuel recovery efforts and planning for the future. For months, researchers have said that the government’s official count of 64 from the September storm was an undercount. The Harvard Study, published in the New England Journal of Medicine a few days before the start of the June 1 hurricane season, has attracted a lot of attention.

But it’s not the only study that’s worked to come up with a more accurate death count; there are actually several studies that have tried to create their own estimates.

"There is growing body of literature and growing scientific consensus that it is not 64," said Mario Marazzi-Santiago, director of the Puerto Rico Institute of Statistics. "The next question is what is the right number, and that will take some time."

Part of a death toll is a count of direct deaths from a storm, such as being struck by a falling tree or drowning in a flood.

The more complicated count comes from indirect deaths caused by unsafe or unhealthy conditions, such as a lack of electricity cutting off a dialysis machine to a kidney patient.

Usually, the death toll of an event such as Hurricane Maria is determined through an exam by a medical examiner. But Puerto Rico officials did not have the proper resources to effectively conduct forensic examinations.

Without complete medical records, researchers were left with other ways to try to come up with an estimate.

Researchers from Harvard T.H. Chan School of Public Health relied on interviewing people and asking them about causes of death in their households. Researchers selected 3,300 randomly chosen households and found 38 deaths after the hurricane, including three from direct causes and 12 from interruption of necessary medical services. The Harvard researchers extrapolated based on that data about the number of "excess deaths" (the number of deaths compared with the same period during the previous year) and found a 62 percent increase in the mortality rate.

The researchers concluded that there was a range of 793 to 8,498 deaths with a confidence interval of 95 percent. But it was that midpoint number of 4,645 "excess deaths" that drew most of the media attention.

In a question and answer document, the researchers emphasized that there is uncertainty associated with the estimate. The researchers said that they didn’t say that 4,645 died: "We provide a 95 percent confidence interval of 793 to 8,498, and 4,645 falls in the middle of this range." (Marazzi-Santiago of the Puerto Rico Institute of Statistics said that the confidence interval means that if samples were taken over and over, 95 percent of the samples would lead to an estimate within the range of about 800 to 8,500.)

We searched the Nexis news database and found more than 100 news articles with the figure of 4,645 or 4,600 in the headline, but far fewer explained that the study actually cited a range.

University of Puerto Rico statistician Roberto Rivera, who along with colleague Wolfgang Rolke used death certificates to estimate a much lower death count, said that indirect estimates should be interpreted with care.

"Note that according to the study the true number of deaths due to Maria can be any number between 793 and 8,498: 4,645 is not more likely than any other value in the range," Rivera said.

Rivera published an analysis in February concluding that in the first six weeks after Hurricane Maria, the death count was between 605 and 1,039.

Following the Harvard study, the government in Puerto Rico released additional data showing there were at least 1,400 additional deaths on the island in the months after the hurricane. But the government didn’t explain how many were attributed to the hurricane.

Other researchers have also estimated a death toll in the ballpark of 1,000, including the New York Times.

Alexis R. Santos-Lozada of the Pennsylvania State University was part of the team of demographers in 2017 that found that there were 1,085 added deaths. Santos-Lozada told PolitiFact that his finding doesn’t conflict with the Harvard study when examining what it said in full: an estimated range between 800 and 8,000.

"The Harvard study has a range that includes my value within it," he said.

Donald Berry, a biostatistics professor at the University of Texas MD Anderson Cancer Center in Houston, has been one of the biggest critics of the Harvard study. He said that the study compares apples to oranges by comparing actual death rate data in 2016 with data based on interviews in 2017.

"The big thing is the methodology is so completely different, you don’t know what you are dealing with," he said. "What you end up with is garbage."

Months ago, the government in Puerto Rico commissioned a study to examine the death toll by the George Washington University Milken Institute School of Public Health. That study is ongoing.

While the Harvard study was based on interviews with households, the Milken Institute study will examine death certificate and other mortality data from September 2017 to the end of February 2018. In a statement, it said it expected its study to produce "a narrower range of uncertainty."

The first phase of the study is expected to be done this summer.

On June 1, Gov. Ricardo Rosselló told CNN’s Anderson Cooper he welcomed the results of the Harvard study and does not stand by the official death toll. He emphasized the importance of understanding that that original count was far too low.

"We had established that this number was going to be much, much higher than what we had as an official tally," he said.

UPDATE Sept. 13, 2018: Since this story posted, another study attempting to approximate the death toll was commissioned by the government in Puerto Rico and published by the George Washington University Milken Institute School of Public Health. The study released in August 2018 analyzed death certificates and other mortality data and found an estimated 2,975 excess deaths between September 2017 through the end of February 2018. The team compared the total number of deaths during that time to the expected number based on historical patterns and found that the number was 22 percent higher than would have been expected.