Pandemic Tracking and the Future of Data

Roman Mars:
This episode is one in a four-part series we’re calling the “The Future of…” We’ll be exploring how changes to the way we live, learn, work and play may shape our health and wellbeing in years to come. Thanks to the Robert Wood Johnson Foundation for supporting this episode. The Robert Wood Johnson Foundation is committed to improving health and health equity in the United States. Learn more about them at rwjf.org.

——————

Roman Mars:
This is 99% Invisible. I’m Roman Mars.

[MUSIC]

Delaney Hall:
Hey, Roman!

Roman Mars:
Hey producer Delaney Hall! So you’re here with the next story in our series called “The Future of …” And you’re here to talk about the future of … data.

Delaney Hall:
Yes, that’s right – and I feel like I should just level with you and our listeners. This is a story about spreadsheets, okay? It’s a story about spreadsheets and data entry and data systems.

Roman Mars:
– A story about spreadsheets, it’s not even my birthday, Delaney! How did I get this marvelous gift?

Delaney Hall:
I should’ve known I wouldn’t have to convince you.

Roman Mars:
You do not have to convince me, I’m already riveted. So let’s go.

Delaney Hall:
Okay, we’ll start in the early days of the pandemic.

NEWS CLIP
[TODAY THE ENTIRE CITY OF WUHAN IS ON LOCKDOWN …]

Delaney Hall:
And as COVID spread in Asia, and then in other parts of the world –

NEWS CLIP:
[THE MOST ACTIVE HOTSPOT NOW IS ITALY… ]

Delaney Hall:
– you were seeing officials take these wildly unprecedented steps to control the disease – you know lockdowns, quarantines, massive amounts of testing. And I think a lot of people thought that we’d see the same kind of response in the U.S., when the virus arrived here.

Alexis Madrigal:
You know, as someone who grew up reading “The Hot Zone,” I expected that we had the world’s foremost infectious disease fighting agency in the world.

Delaney Hall:
This is Alexis Madrigal – at the time the pandemic started, he was a journalist at “The Atlantic.” He’s still a contributing writer there.

Alexis Madrigal:
And so I was expecting to see the United States of America assume global leadership and come up with ways for us to deal with it.

Delaney Hall:
And like Alexis, I was basically imagining scenes from “Outbreak” and “Contagion” in my head. You know, brave epidemiologists in hazmat suits working with the best technology available to get a scary situation under control. And so, of course, we all wanted that to be the reality – we all wanted that to be what our pandemic response looked like. Here’s Robinson Meyer, Alexis’ colleague at the Atlantic.

Robinson Meyer:
Right, the CDC seemed like the last bastion of, like, absolutely competent American technocracy. Right? Like they were on top of it.

Delaney Hall:
BUT as we started to see cases of COVID here in the U.S., Alexis and Rob felt like the response from our public health agencies was surprisingly muted. We were NOT seeing mass testing like was happening in Asian countries – and in fact when they started looking it was hard to find any concrete numbers at all about how much testing was happening here – which was not good – because testing was the most crucial data point we had for understanding the pandemic at this point.

Roman Mars:
So I guess the CDC wasn’t tracking that at the time?

Delaney Hall:
Well at first it was, but in early March 2020, the agency stopped reporting the total number of nationwide tests. And basically, they said most testing is happening at the state level. So if you want to know, go ask the states –

Robinson Meyer:
The CDC just wasn’t saying. So I was like, fine, we’re going to go to the states.

Delaney Hall:
So they reached out to all the states and asked some really basic questions – how many people have you tested, how many positive cases have you had, and how many tests can you do each day? And they were shocked by what they found.

Alexis Madrigal:
Oh, man, we tested almost nobody!

[MUSIC]

Roman Mars:
What’s that mean: “almost nobody”? Like – what are the numbers for almost nobody?

Delaney Hall:
Well, on March 6, which was just a few days after CDC stopped reporting testing numbers, Rob and Alexis could only verify that 1,895 people nationwide had been tested for COVID.

Robinson Meyer:
– even though the White House was saying that tens or hundreds of thousands of people could be tested per day by that point.

Roman Mars:
I do remember some of this. There was a huge shortage of tests for hospitals to use at the beginning of the pandemic and there were pretty strict rules by the CDC about who could get tested at all because tests were in short supply.

Delaney Hall:
Yeah, and that ended up having enormous implications. Because without much testing, it was hard to know what was happening on the ground.

Robinson Meyer:
If you looked at the state of the data, there was no data! Many states could test only a dozen people a day. If the virus was circulating, we had no way to know.

Delaney Hall:
And Alexis and Rob felt like they’d wandered into this weird twilight zone, where people at the top of our public health agencies were talking about carrying out a data-driven pandemic response but without actually having much data.

Robinson Meyer:
And you’d be like, well, it seems like the virus could be everywhere then. And they would be like, well, the data doesn’t say it is. But the data was [beep]!

Delaney Hall:
And it wasn’t just the testing data that was limited. It was also hard to find numbers on how many people were hospitalized and how many people had died. And so Rob and Alexis decided to team up with a guy named Jeff Hammerbacher, he’s a scientist and software developer. And they started compiling their own nationwide data sets, based on what states were publishing. And they thought of this as a stopgap.

Alexis Madrigal:
These are not fancy things, right? Cases, hospitalizations, deaths, number of tests performed – like this is basic stuff. Like you look around the world and like all these other places, just like have the stuff, you know. So we were just like, well, obviously we’ll also soon have this stuff.

Delaney Hall:
Because how could you run a public health response without good data?? It’s like the lifeblood of public health. And it has been since the beginning of the field.

[MUSIC]

Delaney Hall:
Okay, before we continue with the saga of Rob and Alexis, I want to tell you about some of that early public health history – because some of its fun, honestly that’s a big reason. But also because it shows why data is so important and why our system makes it so hard to collect.

Roman Mars:
Okay, let’s do it.

Delaney Hall:
Okay, so there were many innovators in the realm of health data – you know, people like John Snow, William Farr, WEB DuBois – but I’m going to zoom in on one of the earliest examples. We’ll start with John Graunt – and the Bills of Mortality.

Roman Mars:
The Bills of Mortality. I like this. This is very ominous! I like this already.

Delaney Hall:
The Bills of Mortality were these mortality reports that were published in London in the 1600s. And every week, a group of parish clerks would gather and share information about who had died in their neighborhood and how.

Steven Johnson:
And people would read them – they’d kind of pick them up at the local parish thing and be like, “Oh, you know, Joan died.”

Delaney Hall:
This is Steven Johnson, he wrote a book called “Extra Life: A Short History of Living Longer.” And he says the Bills were a source of fascination for many people – people liked to gossip about them – but they weren’t what you would call “structured data” – like there wasn’t really a way to detect useful patterns in the information there.

[MUSIC]

Delaney Hall:
But then John Graunt comes along. And Graunt was a haberdasher/amateur -demographer.

Roman Mars:
That’s a very 1600s combination of interests!

Delaney Hall:
Yeah, you don’t get that kind of mix anymore. He was a classic kind of engaged amateur and he looked at the Bills of Mortality and thought, “There is more to be learned from these documents.”

Steven Johnson:
— and so it occurred to him at some point in the early 1660s that if you tried to assemble that data and look at it systematically that you might be able to tell a lot more about what was really happening to the health of Londoners… And so he spent all this time going from kind of parish to parish and reading through all these reports and kind of tabulating basically the data and trying to organize it in a more structured way.

Delaney Hall:
And Graunt ended up publishing a pamphlet with a very catchy title – it was called “Natural and Political Observations Mentioned in a Following Index and Made Upon the Bills of Mortality.”

Steven Johnson:
And what it enables both Graunt to do, but also health officials around the city, is to suddenly be able to answer the question, what is really killing people? You know, where are the real threats and how are those threats changing over time?

Delaney Hall:
Graunt’s work represented a huge conceptual breakthrough. His pamphlet was basically the founding document of medical statistics and public health data. But it ultimately wasn’t that useful. Because there was still a limited understanding of what was actually killing people. For example, let me share with you the “causes of death” tabulated in this report –

Steven Johnson:
This list is pretty funny. According to Graunt, in 1662, about thirteen hundred people in London died of apoplexy. Thirty eight died of cut of the stone, 74 died of falling sickness, 243 died of “dead in the streets.” Only six died of leprosy…. You’d think there’d be more in 1662. 158 died of lunatic… And then my favorite category, 454 died of “suddenly.”

Roman Mars:
You can totally imagine the scenario where you get the answer “suddenly.” Some official comes up and asks, “How’d he die?” Someone says, “Suddenly!”

Delaney Hall:
Totally, yeah, I wonder if “gradually” shows up in the tabulations?

Roman Mars:
Well then I can see, you know, if bad data comes in, there’s not much you can do about that. But I can still see how it’s groundbreaking.

Delaney Hall:
Yeah it was groundbreaking in that it took these anecdotes and turned them into data about a population, but it wasn’t… actionable. Partly because of answers like “suddenly” – but also because – to act effectively on that data, they needed institutions that could coordinate a public health response – and those did not really come into being – at least in the U.S. – until the middle of the 19th century.

[MUSIC]

Delaney Hall:
In the 1800s, a number of cholera epidemics swept across the United States. And in response, cities and states across the country started to establish local public health departments. But the system here grew in a mostly bottom-up way – it was cities and states first. It took decades before any real national public health institutions came along.

Roman Mars:
Huh. So when did the federal government begin to get more involved in public health, like across the whole nation?

Delaney Hall:
Pretty late in the game! There was one national agency, called the Marine Hospital Service, which cared for sick and injured sailors. Eventually, in 1912, that agency became the Public Health Service.

David Rosner:
It started taking on greater powers, but it was still very hesitant to interfere in anything that was local.

Delaney Hall:
This is David Rosner. He’s a historian of public health at Columbia University.

David Rosner:
The federal government is basically the weakest part of the national structure. This was, you know, not a country that saw the federal government as much more than a bunch of buildings in a swamp in the D.C. area.

Delaney Hall:
Over the next few decades, the national system grew. The CDC eventually emerged from the Public Health Service and became, in many ways, the best disease fighting agency in the world. They pioneered the whole idea of a data-driven response, you know, using stats to figure out who was being hit by disease and how to intervene.

David Rosner:
It’s this moment when a lot of American culture begins to turn to technology and science in general as a means of addressing all these very sticky social problems that plagued us. And for that, you needed data. You needed some sense of what society looked like so if you saw, you know, statistics that showed high death rates in one community or another, you could begin to rationalize your resources and identify resources.

Delaney Hall:
And the CDC had some huge successes. The agency helped eradicate smallpox, it started the fight against HIV, it stopped Ebola – more than once. Over time it developed a heroic reputation…. But there was always this underlying weakness in our system, which was that it’s very fractured. It wasn’t a coordinated system like some countries have. Instead, our system is this patchwork of thousands of state and local health departments who all operate fairly independently. So the CDC can issue guidance, but ultimately state and local health departments answer more to their local elected officials than they do to the CDC

Roman Mars:
Yeah, it sounds like federalism as a concept is a real challenge to public health – like a lot of power resides in states and it’s made that way on purpose but when it comes to public health, these things don’t have state boundaries – a flu or COVID can pass through state boundaries and doesn’t care about federalism at all as a concept.

Delaney Hall:
Right, viruses do not care about states rights, they do not care about state jurisdictions. Over time, there was another damaging pattern that began to develop with the CDC – and the public health system more widely – which was that it struggled for consistent funding. So when there was an immediate crisis, there would be an infusion of cash. But then, when the crisis had passed, the resources would evaporate. And that only accelerated from the 1980s onwards during the Reagan era.

Roman Mars:
Yeah, Reagan is at the start of many of these things when it comes to the social safety net being eroded. This is a familiar story for a lot of agencies.

Delaney Hall:
Yeah, and this is a complicated part of the story that we’re not going to wade super deep into, but to simplify, public health agencies saw their budgets get cut decade after decade all the way into the 2000s. The CDC’s budget dropped overall from 2010 to 2019. Over the same time period, local public health departments lost more than 50,000 jobs due to funding cuts. And we also saw a ton of privatization during this time – so the hiring of private contractors to do what the government used to do.

[MUSIC]

Delaney Hall:
And so this was the under-funded and very complicated system that Robinson Meyer and Alexis Madrigal encountered, when they set out to gather their own COVID related data at the start of the pandemic. Here’s Alexis.

Alexis Madrigal:
The federal government itself, like doesn’t actually have the people to do the things that other governments do, because we just decided, from Reagan onwards, that we were essentially going to take out state capacity and instead pay consulting fees and contracts to companies, because I guess somehow that’s like less government-y or something?

Roman Mars:
And I imagine all of this makes data-gathering and sharing really tough.

Delaney Hall:
Yeah, it’s kinda hard to imagine a system worse than this one when it comes to data sharing. All these entities report data on what’s happening locally, often a little bit differently, using different systems and data conventions.

Alexis Madrigal:
You just have so many reporting entities and officials and jurisdictions. And it’s not just because we’re a large country. It’s also because we have built these data systems over time in this kind of sedimentary pile. And it’s very difficult to change them when it’s not a crisis. You know, like it’s just hard.

Roman Mars:
And so did Alexis and Rob fully understand the complexities of the health system when they started to gather these nationwide statistics about COVID?

Delaney Hall:
Not really. Like they just knew that the government wasn’t releasing much information about what was going on – and so they started their own DIY effort and recruited volunteers to help out with it.

Erin Kissane:
– and it turned out I was literally the first person to fill out the volunteer form.

Delaney Hall:
Erin Kissane has worked on a bunch of web-focused projects over the years, most recently as an editor and director of content with Knight-Mozilla OpenNews. And she had been following COVID really closely and just jumped right into the data-gathering.

Erin Kissane:
And the work immediately was just things like, “Hey, can someone set us up a website?” And at the same time, like, how do we open up this spreadsheet and get it to hold up with 20 people going and collecting data? And what kind of quality processes should we set up because we obviously need to have people double checking these numbers? So it was sort of everything all at once.

Delaney Hall:
This group came to be known as the COVID Tracking Project. And full disclosure, the COVID Tracking Project received funding from the Robert Wood Johnson Foundation, which also funded this story. Within a few weeks, the COVID Tracking Project had several hundred volunteers helping out. There were students, there were people from tech, people from medicine, journalists –

Roman Mars:
So what exactly did the work they were doing look like? How do you begin to gather it?

Delaney Hall:
Well, this was very simple and very complicated at the same time. Every day, they would reach out to every state and territory in the U.S. to find out how many tests they’d done, and numbers of positive cases, hospitalizations, and deaths. And they used a range of methods to get those numbers. They watched a ton of press conferences, for example – and they would also scrape numbers from state websites – then they’d put everything into a spreadsheet –

Alexis Madrigal:
And then, once a day, you sort of commit those numbers and you’re like, “Okay, this is the day’s numbers.” And it is easy to do one time and it’s easy to even to do, you know, 20 times.

Delaney Hall:
But as time went on, it started to get much harder. Because they began to understand the idiosyncrasies of the system – and also the data itself. For instance, take the most basic unit of this data: a case of COVID. Like I would think that would be pretty easy to define and count, right?

Roman Mars:
Yeah, yeah totally.

Delaney Hall:
Well, it is not!

Erin Kissane:
We had to sort of figure out over time that a case is not a case. In some places, a case is a confirmed case. In some places it’s confirmed and what’s called probable cases. Well, what are those? Okay, here’s the definitions. Is this state combining confirmed and probable cases? We can’t tell. Let’s call the state!!

Roman Mars:
This is making my head hurt just thinking about it.

Delaney Hall:
I know, and this was true for most of the metrics they were trying to track.

Alexis Madrigal:
There were all of these different ways that what the states were sending the federal government was just slightly different. You had different data definitions, you had different systems. And no one really had any idea of how to standardize those things, particularly in the midst of a crisis like this.

Delaney Hall:
And there are just so many examples of how our COVID data was unstandardized or incomplete. Like another example, did you know a lot of our local health departments are still at the mercy of fax machines??

Roman Mars:
Oh my god, well I can’t say I’m surprised but I’m still appalled.

Delaney Hall:
It caused ALL kinds of problems! Some states only reported electronically submitted lab results. Others combined electronic submissions and faxed submissions. But because people had to enter the faxed information into their tallies manually – there’d be these big distortions in the data, where all of sudden, a backlog of faxed results would just be dumped into the numbers at once.

Roman Mars:
Right, you hold all the faxed ones til the end and you enter them and the numbers leap – and it makes people mistrust data, is what it does.

Delaney Hall:
Or, here’s another example, when there would be a bad surge somewhere and lots of deaths, the people who fill out death certificates would get really behind – and so death numbers would lag. Or – yet another example – really critical demographic data would just be missing.

Erin Kissane:
There really were whole areas, especially race and ethnicity, where most states and territories never developed a really robust way to collect that kind of demographic data. And when they did, it wasn’t standardized.

Delaney Hall:
– which of course, meant it was hard to tell who was being most impacted by the disease. And this stuff can feel kinda dry, like ultimately we’re talking about spreadsheets and data standardization. But there was stuff going on at this time, early in the pandemic, where really good data could’ve helped.

NEWS ARCHIVAL TAPE
[THESE N95 RESPIRATOR MASKS IN PARTICULAR ARE IN HIGH DEMAND AND SHORT SUPPLY.]

Delaney Hall:
Remember, for example, there was a massive shortage of protective gear for healthcare workers.

NEWS ARCHIVAL TAPE:
[I HAVE SPOKEN TO HEALTH CARE WORKERS IN SAN FRANCISCO, OAKLAND AND SAN JOSE TODAY, ALL WHO SAY THE SHORTAGE OF SUPPLIES IN THEIR HOSPITALS IS A PROBLEM.]

Alexis Madrigal:
If we’d had the hospitalization data – knowing who had PPE, who had access to, you know, medications, who had staffing shortages – the federal government can, in fact, step in and help with that stuff. Instead, that became this weird, crazy scramble that benefited nobody, as far as I can tell.

Roman Mars:
So was the U.S. government… doing this type of data collection from the states and just not reporting it? Or were they just not doing it all.

Delaney Hall:
Well, in those early days, the people at the COVID Tracking Project certainly thought the government had its own comprehensive numbers. They thought that someone… somewhere… within the vast CDC bureaucracy had stats – and you know, BETTER stats – than they did.

Erin Kissane:
I think all of us really thought that the data did exist and we just couldn’t see it. So the work that we were all doing was a stopgap and presumably we would need to do it for a little while until the federal government released the numbers they had.

Delaney Hall:
But then they started noticing some weird… coincidences. This would’ve been in March and April 2020. And back then, there were these press conferences that Vice president Mike Pence was doing –

CORONAVIRUS TASK FORCE ARCHIVAL TAPE
[VP MIKE PENCE: THANK YOU, MR. PRESIDENT. AND LET ME ECHO YOUR WORDS ABOUT ALL THE DEDICATED MEN AND WOMEN ON THE WHITE HOUSE CORONAVIRUS TASK FORCE.]

Delaney Hall:
And the COVID Tracking people would watch those press conferences to see what kind of numbers the government shared — and how they compared to the numbers that they were gathering.

CORONAVIRUS TASK FORCE ARCHIVAL TAPE:
[VP MIKE PENCE: IT WAS REPORTED TO US THAT AT THIS MOMENT, MORE THAN 746,000 AMERICANS HAVE TESTED POSITIVE FOR THE CORONAVIRUS. AND FORTUNATELY, MORE THAN 68,000 AMERICANS HAVE FULLY RECOVERED, BUT SADLY, MORE THAN 41,000 AMERICANS HAVE LOST THEIR LIVES TO THE CORONAVIRUS.]

Erin Kissane:
And we were like, “Hey, you know, his numbers are really close to ours.” We must be doing a good job because he’s getting those federal numbers, clearly. And, you know, here we are, just scraping from public data –

[MUSIC]

CORONAVIRUS TASK FORCE ARCHIVAL TAPE:
[VP MIKE PENCE: THERE HAVE BEEN MORE THAN 843,000 AMERICANS WHO CONTRACTED THE CORONAVIRUS, AND WE GRIEVE THE LOSS OF MORE THAN 47,000 OF OUR COUNTRYMEN.]

Erin Kissane:
– and then we realized they were tracking reeeeally closely. And I think a few of us started to suspect at that point that he was actually reading our numbers just rounded.

Roman Mars:
Wow. WOW. Okay.

Delaney Hall:
And THEN… their suspicions were later confirmed when the Trump administration published a report that clearly used the project’s data and charts, and cited them in the footnotes.

Robinson Meyer:
And it was like, “Oh, they’re just looking at our site” – like we are the process, we are the ones who are making this data.

Alexis Madrigal:
We were waiting for the cavalry and then it turned out like, we were the cavalry and we were like, “No, no, no, no! We don’t even have horses. We can’t be the cavalry.”

Delaney Hall:
And it’s kind of darkly funny. But it’s also scary.

Alexis Madrigal:
You kind of think sometimes like, well, you know, if disaster X were to happen, well, you know, somebody’s thinking about that, you know what I mean? Like, somebody’s gonna do it. And the truth is, no, nobody is going to [bleep]-ing do it sometimes — we were not ready. We did not have a system in place.

Roman Mars:
And so I could imagine them feeling panicked about like, “Oh my god–okay, we thought we were messing around, we’re doing our best. We thought the government was going to save us and they’re not.” But was part of them proud that the government was using their numbers?

Delaney Hall:
I don’t think so, no. Like I’m not sure how everyone on the project felt – but at least some of them were pretty shaken by this realization – like Erin.

Erin Kissane:
That’s an immensely stressful position to be in for a bunch of volunteers. Because on one hand, yeah, it’s great that our numbers are actually really useful and on the other hand, are you kidding me?! That’s the best you can do with the entire resources of the federal government? Is get the data that we make every day by looking at websites??

Roman Mars:
Hmm, I mean, it’s hard for me to understand what was going on at the CDC – like what was going on within the agency at this time?

Delaney Hall:
Well, I’ll start by saying that we came into the pandemic with data systems that just were not designed to gather and process the kind of fast, high-resolution data that people wanted – like the demand for data just went way beyond what public health officials had ever encountered before and they were caught flat-footed. There was also the general organizational chaos of the Trump administration, which certainly didn’t help. But there were a number of ways the CDC tried to compensate. So as an example, in April 2020, the agency started working on a new electronic reporting system that would collect detailed testing data from every state. It took a long time to get all the states onboarded to that system – like more than a year. As that was happening, the agency was also using some of its existing surveillance systems – and methods – to track this new disease.

CORONAVIRUS TASKFORCE ARCHIVAL TAPE
[DEBORAH BIRX: FINALLY, CDC QUIETLY LAUNCHED A NEW WEBSITE…]

Delaney Hall:
So for example, this is another press conference held by the coronavirus task force – in early April – where they talk about data the CDC has started to release –

CORONAVIRUS TASKFORCE ARCHIVAL TAPE
[DEBORAH BIRX: THIS SURVEILLANCE DATA IS BRINGING TOGETHER OUR INFLUENZA-LIKE ILLNESSES WITH THEIR SYNDROMIC MANAGEMENT DATABASES.]

Delaney Hall:
And just to parse that for you –

Roman Mars:
Yeah, that would be helpful –

Delaney Hall:
– so what they were doing is that the agency had adapted a couple of their existing systems for this new task of tracking COVID – their system that tracks unusual levels of disease in places like emergency rooms and urgent care centers — that’s the syndromic surveillance data – and their flu reporting systems. And the way the CDC tracks flu is that they sample the population and then model a broader picture. So the data isn’t comprehensive. And there are definitely some reasons for using the older system. For one thing, the states were already used to it.

CORONAVIRUS TASKFORCE ARCHIVAL TAPE
[DEBORAH BIRX: THE STATES ARE USED TO USING THIS SYSTEM. IT’S IN EMERGENCY ROOMS, IT’S IN HOSPITALS, IT’S IN DOCTOR’S OFFICES. AND IT GIVES YOU INSIGHT…]

Delaney Hall:
But the fact was, these existing systems just weren’t working that well. The agency was struggling to keep track of testing and case rates across the country. It was struggling to update hospital data, which includes critical stuff like bed availability and ventilator supply. And with hospital data, this is like a whole other story, but the CDC was moving so slowly that eventually the agency that oversees them – HHS – just took over gathering those stats and built a much better and faster system

Roman Mars:
But the CDC still seems to be at the center of all this today – so does that mean that they eventually started gathering their stats themselves in a different way or updated their systems?

Delaney Hall:
Yeah, they eventually pivoted but it took months before they started aggregating and sharing more of their own data. So they released their own data tracker in early May. Which was about 15 weeks after the first reported case of COVID in the U.S. and more than 8 weeks after the launch of the COVID tracking project. And even when they did that, there continued to be problems with the data and big discrepancies between their testing numbers and state numbers.

Roman Mars:
And is there a sense of why it would take so long – I mean, like, a few enterprising journalists and a bunch of volunteers had something very quickly. Why do you think it took the CDC so long?

Delaney Hall:
Well, I have reached out to the CDC a number of times to try and get their take on all this – and they haven’t responded. But I think a lot of critics of the CDC think there is something in the structure and culture of the agency that keeps them from moving fast and breaking protocol in an emergency.

Alexis Madrigal:
You know, I think there was an attitude among a lot of people in the CDC about not overreacting to COVID like basically like, “Oh, well, you know, if we design all these systems around the disease de jour…” They even have a comment like this on the CDC’s data modernization page for one of the conferences they had, you know, basically like quoting someone at the conference saying, like, “We can’t like over respond to the disease dejour.” And I’m like, “Oh, I’m sorry, did your other diseases de jours kill 600,000 people???”

Delaney Hall:
It’s closer to a million now.

Alexis Madrigal:
Maybe we should be overreacting to this one. Seems reasonable to design things around this, you know, and I think that that was really a big piece of it – was like they didn’t want custom design systems just for COVID. That was not what CDC wanted to do.

Delaney Hall:
But in an unprecedented situation – with a new disease we’d never encountered before – moving through the population in ways that we were only beginning to understand – we needed new systems. And we needed the public health establishment to be as deep in the data as the COVID Tracking Project was.

Robinson Meyer:
If you’re not in the data every day or every few days, if you don’t know how it’s constructed, you don’t understand what’s actually happening and like where the future hotspots are and the future places to be concerned, and you don’t understand what it looks like when a state explodes with cases.

[MUSIC]

Roman Mars:
So I know that the COVID Tracking Project no longer exists. So how did they make the decision to end the project?

Delaney Hall:
Well, you know, about a year into the pandemic, as Biden was coming into office, vaccinations were happening and the pandemic seemed to be on the wane…and so that was when they stopped it. And part of it was that it was taking a toll on the people involved. They’d never intended it to be a long-term project – and even something as mundane as data entry had a high cost.

Erin Kissane:
While this was happening, we had family members dying. We had people we knew who were in those statistics.

Delaney Hall:
But there was another reason as well – which is that the COVID Tracking Project founders really thought the government should be responsible for this work.

Erin Kissane:
You know, we did not think that the public health data in widest distribution for the United States in the COVID pandemic should come from volunteer labor.

Delaney Hall:
And the Biden administration had promised to create a pandemic dashboard, which the COVID Tracking Project people were excited about. They even helped advise on a framework for how to do it. But now, more than a year into Biden’s term, that still has not materialized. Even though COVID-related data remains both very critical and quite confusing to understand. And, you know, what made the COVID Tracking Project unique as an organization, was that they were dedicated to researching and explaining all these various flaws and inconsistencies in the data. There were other data trackers and other volunteer data groups, but it was the COVID Tracking Project that was really dedicated to that kind of analysis, which we still badly need.

Erin Kissane:
Data can’t talk, data can’t explain itself, particularly when you’re speaking either to this idea of a general public, but also to reporters, to anybody in media, to people even in government agencies. The data has to be contextualized and explained, and that’s still largely not happening.

[MUSIC]

Delaney Hall:
I should say, all the folks I spoke with at the COVID Tracking Project recognize that there are some things the CDC does really really well –

Erin Kissane:
There are incredible scientists at CDC and the NIH doing this remarkable, world changing work on vaccines and all of these other things.

Delaney Hall:
And the CDC did have some successes with data as well. The electronic lab reporting system the agency helped build has apparently really increased the speed and accuracy of state data coming into the agency. But we’re still nowhere near having the kind of surveillance systems that we’ll need the next time a pandemic happens.

Erin Kissane:
If you talk to pandemic people, you know, like this was like a starter pandemic… I mean, it just could’ve been so much worse and it WILL be so much worse. You know, we know we are going to face worse threats and the thing that I have never seen is any real reckoning in the federal government with what we didn’t do – with the failures to build a real surveillance system.

[MUSIC]

Delaney Hall:
While I was working on this story, I ended up thinking a lot about this thing that Steven Johnson told me. He was one of the historians of public health. I asked him if he’d followed the work of the COVID Tracking Project. And what he made of it.

Steven Johnson:
I had two reactions to the COVID Data Project, which was on the one hand, it seemed scandalous that they had to do the work that they had to do – that should have already been underway.

Roman Mars:
Makes sense. That’s my reaction too.

Steven Johnson:
But two was, there was part of me that was like, these are the heirs to John Graunt, right? The amateur data collector who does it because they perceive something is missing in the system and there’s not a lot of time to lose and they need to get in there and fill in this missing piece. That’s a beautiful tradition in the history of health. And so part of me was moved to see it kick into gear.

Roman Mars:
I mean, I get that, I love that story, and I’m struck by the fact that the John Graunt story is a romantic figure – a person jumping in and filling a need, inventing a whole new field of science at the same time. But the story of the COVID Tracking Project, I’m really impressed by the people who did it – but it does not seem like a romantic story at all. It feels like a tragedy to me.

Delaney Hall:
It is a tragedy! We shouldn’t have to “John Graunt” the pandemic. Not after hundreds of years of public health developments.

Roman Mars:
Right, what we really want is a boring story where a bureaucracy just does its job.

Delaney Hall:
Yeah, competent bureaucrats.

Roman Mars: Here’s to the competent bureaucrats.

Roman Mars:
Coming up next, we’ll talk about some potential fixes for our public health data systems. We’ll hear from a former CDC director, and someone who has thought a lot about who gets erased from our current data, and how to make it better. Stay with us.

[BREAK]

Roman Mars:
Support for this four-part series exploring the future of health and well-being comes from the Robert Wood Johnson Foundation, which is committed to improving health and health equity in the United States. Knowing that the healthy, equitable future we all deserve won’t simply arrive, RWJF is exploring how new technologies, scientific discoveries, cultural shifts, and unforeseen events–like those in today’s story–may shape our lives in years to come. Through these explorations, they’re learning what it will take to build a future that provides every individual with a fair and just opportunity to thrive – no matter who they are, where they live, or how much money they have. Learn more about their efforts at www.rwjf.org

And if you like thinking about the future of things and have a hunch about the future, share it at shareyourhunch.org. I’m going there now… Okay, I’m selecting the prompt “I have a hunch” – I have a hunch that the increasing misery of air travel will cause people to reconsider train travel in the US and it will be more popular than it has been for decades. Check out other hunches and share your own hunch at shareyourhunch.org

Roman Mars:
Alright, I’m back with Delaney Hall and we’re going to be talking about how to fix our public health surveillance systems, which, that sounds really ominous when I say it that way but really, that’s what we’re trying to fix.

Delaney Hall:
Yeah, this is the good kind of surveillance system. This is the surveillance that we want

Roman Mars:
So how do we fix it?

Delaney Hall:
I guess, spoiler alert, I don’t think there’s one clear answer to that massive problem.

Roman Mars:
Yeah, I can’t say I’m surprised to hear that.

Delaney Hall:
But I did speak with people during my reporting who had a range of interesting ideas about how to make things work better – both within the system and with data in particular. And especially with race and ethnicity data which represents one of the biggest failures in our current system.

Roman Mars:
Oh, that’s really interesting. Tell me more about that.

Delaney Hall:
So I will get to race and ethnicity data a little bit later. I wanted to start with one immediate fix that came up in conversation with Alexis Madrigal, of the COVID Tracking Project, and he said it really would’ve helped if the federal government had just been *extremely* clear with states about what information they wanted reported, and how. Because then the data coming from the states would’ve been more standardized, and easier to compare and analyze.

Alexis Madrigal:
Probably the thing that would have made the biggest single difference on a data level would be if the federal government had said basically, on an ultra, ultra, ultra precise level is what we need. Like we need it to come from this system and answer all those small questions.

Delaney Hall:
And this was actually something that the COVID Tracking Project ended up doing, in the absence of really clear guidance from the government. They pulled together their own guidelines and distributed them to states.

Roman Mars:
Oh that’s interesting – but I’m guessing if that guidance was coming straight from the government, it would probably be more successful.

Delaney Hall:
Yeah, totally. The government, believe it or not, has more authority than a volunteer effort – however impressive that effort was. The other thing is, even if the government had been really clear about how it wanted the data reported – that would’ve made things better, for sure – but it wouldn’t have solved the underlying issues with the surveillance system. Those are much bigger and more complex.

Dr. Tom Frieden:
You can’t fix the data system without fixing the broader public health system.

Delaney Hall:
This is Dr. Thomas Frieden. He was the director of the CDC from 2009 – 2017. He also worked as the Health Commissioner for New York City for seven years. And he testified before Congress in March 2021 – so this was around the time the COVID Tracking Project shut down – and he said in that testimony that, quote, “Lack of accurate, real-time information was one of the greatest failures of the U.S. response to the COVID-19 pandemic.”

Roman Mars:
Wow okay, that’s definitive.

Delaney Hall:
Yeah, I mean, he said our data systems are broken, basically from the bottom to the top. And he said it’s not just the CDC’s fault here.

Dr. Tom Frieden:
So I think saying, well, CDC couldn’t get the data together… CDC was dealing with local and state health departments that were overwhelmed and couldn’t collect the data, hospitals that didn’t have standardized data, and laboratory testing that was insufficient, and a contact tracing system that never really worked effectively in most places.

Roman Mars:
Wow, I mean, that is a wide range of failures. I imagine this all goes back to both the fractured nature of our public health system and the way states and the federal government really don’t work hand in hand. And then also the hollowing out of these institutions and systems that has been going on for decades.

Delaney Hall:
Right, our system got to this very bad point thanks to under-funding it for decades. And we talked a little bit about this in the piece, we’re already seeing the cycle of panic and neglect, as it’s known, kick in yet again. So a crisis happens, then money and resources pour in. The crisis fades, and the money goes away. And this is just not a good way to fund a system that needs to rebuild some of its critical infrastructure from the bottom up.

Dr. Tom Frieden:
You can’t build a sustainable system with one time dollars.

Delaney Hall:
And he really wants to see us build that sustainable system. He knows – everyone in public health knows – that this kind of real-time data-collection is important. But Dr. Frieden says it’s going to take years of investment to fix it. You know, there needs to be an agreement about a national architecture for data gathering and sharing – the government needs to be able to hire really talented programmers – we need to find workarounds for some very tricky problems, including the fact that we don’t have “national health identifiers” in this country, which means that tracking people across different systems is a big challenge. There’s a lot to sort out.

Dr. Tom Frieden:
We need a multi-year investment to modernize it. It’s not just a matter of replacing fax machines with an electronic secure interchange.

Delaney Hall:
But as we also heard in the main story, this isn’t just about money or technology. Those things are important, for sure. But there are elements of the CDC’s current culture – and how it interacts with local health departments – that also needs to change.

Roman Mars:
Right, so my impression of the CDC is that it’s a very scientific and academic organization in terms of its outlook. Like, they do very careful analysis before releasing or recommending anything, and that’s not always what public demands.

Delaney Hall:
Yeah, I think it’s safe to say that the agency tends to move slow. It’s also, Frieden pointed out, sometimes subject to political oversight and vetting that can contribute to that slowness. But for whatever reasons, I think it moves at a different pace from people in local health departments who are frontline responders. They need to move fast, sometimes with just the best data available at that moment. And so Frieden thinks there should be more movement back and forth between the CDC and local and state-level health departments so they can understand each other’s needs.

Dr. Tom Frieden:
There are too few people working at CDC headquarters in Atlanta who have worked for two or five or 10 years at a county or city or state or global health department – embedded to understand that if you need an answer, sometimes it’s in the next four or five minutes, not in the next four or five hours, certainly not next four or five days.

Delaney Hall:
And so what Dr. Frieden has proposed is having thousands of staff on the CDC payroll who are actually embedded in county and city health departments for a few years, and who then rotate back to CDC headquarters.

Roman Mars:
That’s an interesting suggestion – and are there any indications that the CDC is seriously looking at this, or changing its culture in any way.

Delaney Hall:
I haven’t heard of anything like what Dr. Frieden is proposing, like a cultural exchange between the CDC and local health departments. I also think the COVID Tracking Project founders would say there’s nothing close to the level of soul-searching that they would like to see happening at the CDC right now. Like a real reckoning with what went wrong during the pandemic. BUT there are some new efforts at the agency, like for example, a center within the CDC called the Center for Forecasting and Outbreak Analytics, the CFA –

Roman Mars:
Okay, so what is the CFA supposed to do?

Delaney Hall:
It’s being billed as a “weather service” for disease – a group that can forecast outbreaks. Which is interesting and challenging work. And how it relates to data is that – the quality of any given model, and its resulting forecast, depends very heavily on the quality of the data that goes into it. So in our current system where even simple metrics like test-positivity rates or hospitalizations are ambiguous… that’s gonna be a problem for the pandemic modelers.

Roman Mars:
Totally, if the CFA is going to be successful, they’ve gotta sort out the data stuff from the get-go. Because you can have the greatest model in the world and all these people willing to do it, and you can predict amazing things, but if the data’s not there, then it does not matter.

Delaney Hall:
Yeah. The data stuff has to come first, it’s foundational.

[MUSIC]

Delaney Hall:
And then finally, there’s one other aspect of data that we should talk about because it represents a huge gap in our current knowledge. And that’s how we collect – or rather, do not collect – data related to race and ethnicity. Like if we’re going to be rebuilding our data systems in the way that Frieden is describing, it’s worth thinking through this question in particular.

Roman Mars:
Yeah, I remember when I was following the work of the COVID Tracking Project, this was an issue that they really focused on.

Delaney Hall:
Yeah, it definitely was. The COVID Tracking Project ended up developing a whole wing of their project devoted specifically to race and ethnicity data. They did that work in collaboration with Dr. Ibram X. Kendi, who runs the Boston University Center for Anti-racist Research. And early in the pandemic, Dr. Kendi wrote a series of essays in “The Atlantic” where he argued that we really don’t know who’s being most impacted by COVID-19 – because the data around race is so limited.

Roman Mars:
And sp why is that? Is it that race and ethnicity information is just not gathered? Is it just not shared?

Delaney Hall: The data is insufficient in a number of ways. And to help explain how, I’d like to introduce you to Abigail Echo-Hawk.

Abigail Echo-Hawk:
We all know somebody. We all know somebody who was impacted by COVID-19. We all know somebody who is impacted by a death, even if we weren’t ourselves.

Delaney Hall:
Echo-Hawk is a citizen of the Pawnee nation of Oklahoma and she’s the director of the Urban Indian Health Institute in Seattle, Washington. It’s one of 12 tribal epidemiology centers in the country. And Echo-Hawk says that it’s clear Native people were disproportionately affected by COVID – even just anecdotally, like she said, everybody knows somebody – but that it’s impossible to know just HOW much.

Abigail Echo-Hawk:
Even when data was collected, they weren’t collecting the race and ethnicity of American Indians and Alaska Natives and many other people of color. So while we know the impact on our people was great with the scarce data we had, we know it’s a gross underreporting.

Delaney Hall:
And what’s interesting is that the COVID pandemic has recently focused people’s attention on this issue – like this is something people are now talking about – the fact that the pandemic disproportionately affected people of color. But Echo-Hawk has been interested in the issue for much longer. Because, ever since she started her career in public health, she has seen the ways that Native people – and other people of color – are made invisible in the data. And it happens through a couple mechanisms. One is by virtue of being a small population that can be difficult to gather “statistically significant” data about –

Abigail Echo-Hawk:
I would be in meeting after meeting after meeting where we would be a little asterix down at the bottom that would say “not statistically significant” or “not reported on” – and so what it was, is we were invisible and we were invisible in conversations that policymakers were having, we were invisible when they were allocating resources – and what I saw was incredible health disparities, and the death of community members and family members as a direct result.

Delaney Hall:
So that’s one way the data around ethnicity is lacking. Another thing that happens is that people will sometimes be given options on a form – like maybe Black, white, and “other” – it’s a limited range – and “other” might be the only option that applies so they check that box.

Abigail Echo-Hawk:
And that could include Japanese people. It could include American-Indians, Alaska Natives, it could include other races or ethnicities. And even when you fill that in, they never disaggregate it. That means that they kind of put it all together and they just count that “other.” What that does is it effectively hides what’s happening to a particular population of people.

Delaney Hall:
And this issue of “aggregation” and “disaggregation” is important. Disaggregating the data just means breaking the data down into smaller units or segments instead of bundling a bunch of it up together, which is what happens in the “other” category – it also sometimes happens with people who are multiple races.

Abigail Echo-Hawk:
Somebody like my children who are Mexican-American and also American-Indian, and they mark on a form Hispanic, they mark American-Indian and they mark Filipino because they’re also Filipino. And when the data is calculated, instead, they put them into a category that says multi-race. But they don’t disaggregate it in a way that says they are both Hispanic, they are both Filipino, and they are also American-Indian Alaska native.

Roman Mars:
I mean anyone could look at this and see that “multi-race” is a meaningless category, that probably doesn’t yield very much information at all.

Delaney Hall:
That’s right. It is not a very useful category and what’s interesting is that these other broad categories that we use a lot – like “Asian-American” as an example – when that’s used in public health data, it hides the fact that Asian-Americans are an incredibly diverse group, with very diverse experiences related to health and illness.

[MUSIC]

Delaney Hall:
Finally, Echo-Hawk described yet another way people of color get erased from the data, which is racial misclassification.

Abigail Echo-Hawk:
Racial misclassification is when you go in and instead of asking you what race or ethnicity you are, somebody might look at you and instead check “white” based on visual appearance, check “Black” based on visual appearance.

Roman Mars:
Oh wow, so whoever is filling out the form doesn’t ask – they just make an assumption based on appearance and just fill it in.

Delaney Hall:
That’s right. Echo-Hawk says it happens all the time – and it disproportionately hurts people of color by making the data around their existence and health issues, just fuzzy and incomplete.

Roman Mars:
Does Echo-Hawk have ideas about how to change the way we collect data so this type of stuff doesn’t happen?

Delaney Hall:
She does — she talks about “decolonizing data” – and in addition to basic stuff like disaggregating data and allowing for greater nuance in race and ethnicity categories, she wants to see communities be more involved in deciding what gets gathered and shared about them. So she talks about how there’s a “deficit-based framework” in public health, where in her community, the data always shows high rates of obesity, high rates of diabetes, you know, health challenges. But she also sees a lot of strengths in her community – strengths that can actually measurably improve health – and so she’d like to see data gathered around that too.

Abigail Echo-Hawk:
Yes, we need to know the gaps, but we also need to know – how do our youth see themselves in the future? If you can see yourself in the future, that’s a protective factor against suicidality. We also want to know where their cultural ties are. Are they culturally engaged? Do they have the access to the resources for their cultural engagement? We want to use the strengths of our community, our cultural protective factors, All of those things are things that can be measured and they can be weighted against the gaps.

Delaney Hall:
And her bigger point is just that data should serve the community and the needs of the community. It shouldn’t just be to study the community and write academic papers about it. It should be actionable, and lead to better health for the community it comes from.

Roman Mars:
Well, this is really fascinating and interesting , Delaney. Thank you so much! And full disclosure: like a lot of people who work at the intersection of health and justice, Abigail Echo-Hawk has received funding from the Robert Wood Johnson Foundation, the group that also funded this episode.

———

CREDITS

Roman Mars:
99% Invisible was produced this week by Delaney Hall. Music by Swan Real, sound mix by Ameeta Ganatra. Fact-checking by Graham Hacia. Kurt Kohlstedt is our digital director. The rest of the team includes Vivian Le, Joe Rosenberg, Christopher Johnson, Emmett FitzGerald, Lasha Madan, Jayson De Leon, Martín Gonzalez, Sofia Klatzker, and me, Roman Mars.

We are part of the Stitcher and SiriusXM podcast family, now headquartered six blocks north in the Pandora Building — in beautiful uptown Oakland, California.

You can find the show and join discussions about the show on Facebook. You can tweet at me @romanmars and the show @99piorg. We’re on Instagram and Reddit, too. You can find links to other Stitcher shows I love as well as every past episode of 99pi at 99pi.org.

Post-roll
Thanks again to the Robert Wood Johnson Foundation for their underwriting support of this special episode. Keep an eye out for each episode in this four-part series “The Future of…” And remember, if you have a hunch about the future, share it at shareyourhunch.org