What's Wrong With Our Map Is What's Wrong With Data System

Ideally, anyone should be able to look at this map we created and learn what percentage of health care workers at their local hospitals have not yet been vaccinated against COVID-19.

That may not be possible, however, because some hospitals say the data are incorrect.

The information was directly sourced from the U.S. Department of Health and Human Services (HHS), which compiled its numbers from hospital surveys. So how could the numbers be wrong? The answer to that question is complicated.

In the end, though, the shoddy data mean neither the government nor consumers have enough visibility into vaccination coverage inside the nation’s hospitals, where surprising pockets of vaccine hesitancy persist.

Credit: Georgia Health News

Prior to COVID, the government collected some health information from hospitals, but that data was primarily based on insurance billing. And it was historical information, released months or even years after the fact.

When the pandemic arrived in the United States, policymakers needed a snapshot of conditions inside hospitals in real time, but there was no system set up to collect information on things like hospital capacity, staffing levels, and equipment shortages.

During the Trump administration in March 2020, Vice President Mike Pence, who headed the White House Coronavirus Task Force, sent a letter to hospital administrators across the country asking for daily reports.

The government devised a survey of 32 questions — it has since been expanded — that hospitals are supposed to answer each day. The questions are about things like the number of staffed ICU beds, ventilators, numbers of COVID-positive patients, and COVID deaths.

It’s a wide-ranging survey that requires hospitals to compile information from different departments. Many hospitals, which were already struggling to cope with increased patient loads when the survey was launched, found the data collection onerous.

Flaws and inconsistencies 

There were several false starts. The American Hospital Association said the administration even threatened in June 2020 to send National Guard troops into hospitals to collect the statistics.

Then, in July 2020, Deborah Birx, MD, who was then coordinator for the White House response, wrested control of the data collection system maintained by the CDC and gave it to a politically connected company called Teletracking Technologies.

In August 2020, the government made this reporting one of the Conditions of Participation in the Medicare and Medicaid programs, meaning that if hospitals fall behind or fail to share their numbers, they don’t get reimbursed for their care. Hospitals get millions from Medicare reimbursements annually.

Teletracking began collecting information from hospitals and reporting it to the federal government, which kept the information largely under wraps. It wasn’t publicly released until the Government Accounting Office made a series of critical recommendations urging HHS to make its data collection more transparent.

So now, some of the data is on public view, but it’s not always useful because of all the flaws and inconsistencies.

Some hospital numbers are correct

Based on spot checks done by Medscape Medical News and Georgia Health News, some hospitals have reported their data correctly.

With 49 hospitals, AdventHealth is one of the largest health systems in the nation. At least 20 of its hospitals report fewer than half their workers have been vaccinated against COVID.

Jeff Grainger, a spokeperson for AdventHealth, confirmed that the vaccination numbers reported to HHS were correct as compiled. He added however, that they probably underestimated the true number of workers who had gotten their shots, since the hospital system couldn’t account for cases where workers went off campus to be inoculated, such as at local pharmacies or mass vaccination sites.

Caption

Advent Health in Gordon.

Credit: Brenda Goodman/Georgia Health News

AdventHealth Orlando, the fifth-largest hospital in the United States, according to Becker’s Hospital Review, reported 56% of workers were still unvaccinated as of the week of June 4, 2021.

The hospital said it has surveyed its employees about taking the COVID vaccines, and used that information to shape messaging, but it has not offered bonuses for participation in the vaccination campaign, as some other health systems have.

Grainger said the system requires employees to get flu vaccines and may do the same for COVID vaccines now in use, once they have been fully licensed by the U.S. Food and Drug Administration.

AdventHealth is “an expression of the health ministry” of the Seventh-day Adventist Church, though the church does not operate the hospitals, Grainger said in a statement.

The Seventh-day Adventist Church has issued a statement in support of “responsible immunization/vaccination.”

Confusion pervades data reporting

Other hospitals have misunderstood how to report.

According to HHS data in May, only 4% of the nearly 12,000 people who work with patients at Grady Memorial Hospital were unvaccinated. That includes staff and contractors. But the hospital said that number is wrong.

Instead, 43% of workers at Atlanta’s public hospital, a busy level 1 trauma center, are still unvaccinated.

Grady said it accidentally counted any health care workers it vaccinated — even the ones not working at Grady — in the running total of vaccinated workers at the facility.

On the other end of the spectrum is Geisinger Health System in Pennsylvania. At its hospital in Danville, federal data show 98% of workers are unvaccinated. The hospital said that’s incorrect — it estimates that 28% of employees at that campus remain unvaccinated.

Geisinger, like many hospital systems across the country, reports its numbers to its state health department, which in turn sends the data to the federal government.

The federal government allowed state health departments or hospital associations to become certified to report the information on behalf of their hospitals.

When pressed, a Geisinger spokesperson said she couldn’t explain why the numbers were wrong. She suggested we reach out to HHS and Pennsylvania health officials for more answers.

Amber Liggett, a spokesperson for the Pennsylvania Department of Health, confirmed that it has been collecting information daily from the state’s hospitals and passing it to the federal government.

Health departments are useful conduits because this data allows them to see where problems may lurk at the state and local level. But the Pennsylvania Department of Health isn’t using the vaccination data because it is so flawed.

“We don’t know what data verification is happening at either the facility or federal level; and there currently isn’t any at the state level,” Liggett said in an email.

Another common mistake has to do with the idea of running totals.

Hospitals were asked to report the vaccination data once a week, on Wednesdays.

The numbers are meant to be cumulative. If a hospital inoculated 5 workers in its first week, and then another 5 its second week, the totals reported for weeks one and two should be 5 and 10, respectively, according to this explainer by the CDC.

Some hospitals, including CalvertHealth in Prince Frederick, Md., are reporting their numbers week by week, relying on the government to add them up. Using the numbers in the example above, the hospital’s totals for week 1 and week 2 would be 5 and 5.

CalvertHealth is also moving vaccinated workers into the unvaccinated column if they didn’t get their shots that week.

Let’s say a nurse named Fred works at CalvertHealth. Fred got his second shot of a Pfizer vaccine on Friday, June 4. The next Wednesday, the hospital would report him among its fully vaccinated workers for that week. The following week, though, Fred would get moved to a column tracking unvaccinated workers because the hospital told us that he wasn’t among the workers vaccinated that week.

As a result, it looks like CalvertHealth’s number of unvaccinated workers is growing week by week, while its number of fully vaccinated workers is getting smaller, since they’ve hit a saturation point with vaccinations at their hospital.

When we run our calculation, it looks like nearly all of CalvertHealth’s employees are unvaccinated, but the hospital tells us that’s incorrect — about 30% still have not gotten their shots.

States respond

In other cases, the data is being collected but states are blocking the release of the information.

We approached Houston Methodist Hospital in Texas to find out why its numbers are not in the database. The hospital’s CEO has been outspoken on the issue of worker vaccinations and has been publicly announcing the hospital’s progress.

Houston Methodist was confused, too. It was reporting the numbers, it insisted.

Credit: Georgia Health News

It turns out that while Houston Methodist is sending the data to the Texas Department of State Health Services (DSHS), the state is blocking the release of the numbers.

In fact, only 4% of Texas hospitals have numbers in the system.

Chris Van Deusen, a spokesperson for DSHS, said the agency was withholding the numbers because they were voluntary fields “and the confusing and overlapping nature of the questions could lead to skewed data,” he said.

In Nevada, just 2% of hospitals are reporting the vaccination status of their workers.

Amy Shogren, a spokesperson for the Nevada Hospital Association, admitted that it is not collecting vaccination data from the state’s hospitals for the government’s HHS database because Nevada’s hospitals report that information to a different database designed to collect information on vaccine administration. She declined to say why the numbers couldn’t be reported in both systems.

Keeping workers and patients safe

Vaccination of health care workers matters because it protects these front-line professionals who work in occupations most likely to put them in contact with the SARS-CoV2 virus. More than 3,600 health care workers have died over the course of the pandemic in the U.S., according to a recent investigation by Kaiser Health News and The Guardian.

It is also a patient safety issue. People who catch COVID spread the virus before they show any symptoms. So a health care worker could pass the virus to vulnerable patients before they even knew to stay home from work, especially if they work in a setting where there’s no regular testing program in place.

While someone who is healthy might not experience a severe COVID infection, a patient whose immune system is weakened by medication or a health condition could die.

Houston Methodist Hospital recently announced that nearly 100% of its health care workers were vaccinated after it became the first hospital in the nation to mandate the vaccines. Other hospitals followed suit, after the Equal Opportunity Employment Commission issued a ruling that employers could require the vaccines as long as they offered reasonable exemptions for workers.

But there are still many hospitals in the nation — particularly in rural areas — where large portions of the staff are still unvaccinated. And there’s no way for patients to find out before they walk in.

Letitia Armstrong recently took her husband to Baptist Health in Lexington, Ky., where she reassured the nurse who was treating him that both she and her husband were fully vaccinated, adding, “I’m sure you’ve had your vaccine, too.”

The nurse replied that he had not done so.

When she asked why, he said, “Well, I’m young, and I figure if I get it, I wouldn’t get very sick.”

Armstrong, who runs an elder care company in the area, said she was furious. Because her husband was very ill and the nurse was working on him, she held her tongue.

Baptist Health isn’t reporting data on vaccinations of its health care workers, and it doesn’t have to. Reporting is currently voluntary.

About 48% of hospitals across the country have declined to reveal the vaccination status of their workforces to the federal database, or their state isn’t providing it.

Patient safety experts think the staff vaccination numbers should be required, too.

Reporting of employee vaccination rates “should be mandatory so patients can make a decision [of hospitals] if they have a choice,” said Mark Ebell, MD, professor of epidemiology and biostatistics at the University of Georgia in Athens.

“For someone who is immunocompromised in a hospital, the last thing they need is to get a serious infection,” he said.

Fixing the data

As in many other areas of public health, decades of underfunding and complacency have weakened America’s public health data collection. The pandemic blew holes right through it.

“We weren’t ready for it,” said Charles Rothwell, former director of the National Center for Health Statistics, who retired in January 2019 after three decades at the agency.

Though Rothwell and others had sounded warnings that America’s public health data systems needed more investment, he said he feels personally responsible for not having made a more compelling case to policymakers. “We should have been better prepared,” he said.

Credit: Georgia Health News

Even as the U.S. took weeks to develop and then scale an effective test for the virus, public health agencies were struggling to compile even the most basic numbers on the health emergency.

Journalists and data experts at nonprofits jumped in to fill the gaps with invaluable and heroic efforts like the COVID tracking project and COVID Exit Strategy. The COVID tracking project, started by journalists at The Atlantic, used a network of hundreds of volunteers to compile daily numbers on cases, hospitalizations, tests and deaths. COVID Exit Strategy, which looked at trends in daily cases and tests was helmed by a group of public health and data experts who worked for nonprofits and universities.

“We’re talking about basic data about how many people are in the hospital," said Jennifer Madans, who recently served as acting director of the National Center for Health Statistics. "What are they in for? How many people died? And how much protective equipment do you have?”

She said the U.S. has whole agencies with expertise in how to collect and report information like this, but they were not tapped for help.

Instead the contract was given to Teletracking, which has been populating the HHS Protect Dataset. That’s the data we used to make our map.

So now, the data are on public view, but it’s not always useful because of all the flaws and inconsistencies.

That’s not surprising to Madans, who said the United States historically has spent more money and resources on tracking agriculture statistics than it spends compiling basic health information.

In March 2020, the government allocated $500 million to the CDC through the CARES Act to upgrade its data collection activities.

But experts say more money is still needed to help our health data infrastructure catch up, and efforts are under way to improve the system.

A bipartisan group of lawmakers introduced the Health Statistics Act in February to fix information gaps and barriers that were exposed by the pandemic. The act would allocate another $450 million to the system, as well as standardize health data collection and require the government to share it more broadly across agencies.

And it would require the participation of agencies like the National Center for Health Statistics (NCHS) that have expertise in this kind of data collection.

Madans said the investment is long overdue and has never been more urgent.

She said if NCHS had put out a database like the one we used to make our map, and it was found to be so full of flaws “we would have been just destroyed. It would have been terrible.”

Read part one and part two of this series.

This story comes to GPB through a reporting partnership with Georgia Health News