Why an analysis of COVID-19 vaccines from Florida’s surgeon general is flawed

If you’re a male Floridian between the ages of 18 and 39, recent news about the COVID-19 vaccines might have been alarming. 

Florida Surgeon General Dr. Joseph Ladapo and the state’s department of health released new guidance recommending against the COVID-19 mRNA vaccines for that demographic. They said there was an 84% increase in cardiac-related deaths for vaccinated men in that age group.

"If it had been known two years ago or so that this vaccine would increase cardiac deaths in young men by 84%, would they have approved it? The obvious answer is, 'no,'" Ladapo said during an Oct.10 interview on "Tucker Carlson Tonight." "You would never give something to someone who is young and healthy and increase their risk of dying from sudden cardiac death by 84%." 

But doctors and epidemiologists pushed back, saying the analysis that the recommendation is based on is flawed. 

In the back-and-forth, what should you know about the analysis and recommendation? 

We found that the data in the study is limited and does not paint a clear picture. The study wasn’t peer-reviewed or published in a scientific journal, which means its accuracy wasn’t measured by other scholars. And the new recommendation ignores important context about the vaccine’s safety record. 

Dr. Daniel Salmon, a vaccinologist and director of the Institute for Vaccine Safety at Johns Hopkins University, said the study’s conclusions may sound scary, but don't mean much.

"Firstly, if they did a good study, they should write it up appropriately, with sufficient detail. And they didn't do that. So, posting a report with insufficient detail, flawed data, that didn't consider the vaccine benefits and then taking this 84% and making a big deal out of it. And then saying that young men shouldn’t get vaccinated, it's really irresponsible," Salmon said.

The objective of the health department’s analysis was "to evaluate the risks of all-cause and cardiac-related mortality following COVID-19 vaccination," it said.

"Far less attention has been paid to safety and the concerns of many individuals have been dismissed," Ladapo said when the analysis was released Oct. 7. "These are important findings that should be communicated to Floridians."

The study examined the death certificates of patients who died within a 25-week period after COVID-19 vaccination.

But, doctors pointed out, the number of deaths within the demographic in question was tiny.

"The results for males age 18-39 who received an mRNA vaccine included a grand total of 20 deaths during the risk period," Dr. Kristen Panthagani wrote in her medical blog, You Can Know Things. "If even just a few of these have another cause (which is highly likely), they lose statistical significance." Panthagani is an emergency medicine resident at Yale University who wrote extensively about the Florida analysis on Twitter and her blog. 

Dr. Robert Wachter, the medicine department chair at the University of California, San Francisco, echoed that point in an interview with The Washington Post.

"We’re talking about a very small number of deaths. An extra death or two would potentially change these results," Wachter said.

Another problem with the data is that the 25-week period used in the study was too long of a window, said Dr. Paul Offit, a member of the advisory committee to the U.S. Food and Drug Administration on COVID-19 vaccines and director of the Vaccine Education Center at Children’s Hospital of Philadelphia. 

"Who does a study where you look at the effects 25 weeks later? I mean, if you’re causing an immune response to your heart, that all happens within a week or 10 days," Offit said. "So you should know about that immediately, if there’s a dramatic reaction to your heart."

We reached out to the Florida Department of Health to ask for any additional evidence they wanted us to consider. A spokesperson pointed us to Ladapo’s Twitter thread responding to critics of the analysis, which does not substantively address the issues raised.

Panthagani told PolitiFact that even the process of selecting the death records was questionable. The study selected records with specific ICD-10 codes, which she found concerning. 

ICD-10 is short for International Classification of Diseases, 10th edition, which assigns numbers to disease diagnoses. When choosing which deaths to examine, state analysts selected only a handful of ICD-10 codes related to heart ailments, Panthagani said. ICD-10 codes can be a valid measure to use when examining deaths, but not always, Panthagani said. They don’t fully explain the cause of death. 

The Florida analysis included cardiac-related deaths "if their death record contained an ICD-10 code of I30-I52," it said. That covers conditions including endocarditis, valve disorders, myocarditis, cardiomyopathy, cardiac arrest, conduction system disorders, atrial fibrillation and flutter, heart failure, other cardiac arrhythmias, endocarditis and more, according to Science-Based Medicine. 

"In short," Panthagani tweeted, "this list is too broad to be meaningful, excludes some cardiac issues but not others, and most of the diagnoses are far more likely to be caused by other ongoing disease processes rather than vaccination."

Typically, scientific studies undergo a rigorous process to prove their credibility, Offit said. That involves being submitted for peer review by scholars who look for flaws and confirm whether a study provides adequate proof of the theory being presented. 

"You can’t just stand up and declare something as being true without subjecting it to the normal process of having your data reviewed to see whether or not the data proves a point," Offit said. 

Dr. David Gorski, a professor of surgery at Wayne State University School of Medicine and managing editor of Science-Based Medicine, had the same critique. 

"The authors of the study are anonymous. Its funding source is not disclosed. Its methods section is opaque and makes it hard to figure out exactly how the analysis was done," Gorski said. 

"I could easily publish a study on my blog and claim it shows something, and you would be right not to trust it or take it seriously. I might have taken this somewhat more seriously if the paper had disclosed the authors and stated that it was submitted to a decent peer-reviewed journal for publication. As it is, I just view this as propaganda."

We asked the Florida Department of Health why the study lacked authors’ names and was not peer-reviewed. We did not receive a response. But in an Oct. 11 interview with The Washington Post, Ladapo called it a "fake issue." 

Spokespeople at the U.S. Department of Human Health and Services and the Food and Drug Administration pointed us to previously published guidance, which says it is overwhelmingly more safe to take the COVID-19 vaccine than to risk being unvaccinated and infected with COVID-19. 

Florida’s new recommendation "is flawed and a far cry from the science: COVID-19 vaccines have been proven safe and effective, and severe adverse reactions are rare," Sarah Lovenheim, assistant secretary for public affairs for the Department of Health and Human Services, said in a statement. "The benefits of COVID-19 vaccination — preventing death and hospitalization — are well-established and continue to outweigh any potential risks."

Clinical trials were used to evaluate the vaccine in tens of thousands of participants, and since then, tens of millions have been vaccinated, Lovenheim said. 

No vaccine is perfect. Rare cases of myocarditis, which is inflammation of the heart muscle, and pericarditis, inflammation in the heart’s outer lining, have been reported within several days after vaccination with mRNA vaccines. 

A study published by the scientific journal JAMA Network reviewed data on the vaccine's safety from December 2020 to August 2021. The study found a small but increased risk of myocarditis for men after mRNA COVID-19 vaccines. 

The Centers for Disease Control and Prevention wrote about the study and said that for 18-to-24-year-old males, there were 52.4 cases of myocarditis per million doses with Pfizer and 56.3 cases per million doses with Moderna. Symptoms of myocarditis are normally mild. 

"Most patients with myocarditis or pericarditis after COVID-19 vaccination responded well to medicine and rest and felt better quickly," the CDC wrote. 

Correction, Oct. 17: A previous version of this story included an incorrect title for Sarah Lovenheim, who is assistant secretary for public affairs for the Department of Health and Human Services.