Abstract

This project explores death data compilation history from Chicago’s July 1995 heat wave, framing the heat wave as a case study through which to examine public health data pipelines in the lead-up to the COVID-19 pandemic. By examining decades-old data classification discrepancies, data compilation challenges, and the surrounding landscape of fragmented and aging technical systems, the project argues the history of this information infrastructure is crucial to contextualizing the COVID-19 data we have today.

Background

I began this one-semester fellowship project with three broad questions about Chicago’s COVID-19 data. First, I wanted to better understand where the city’s COVID-19 data originates. Put another way, I wanted a firm grasp of what happens immediately following an individual’s COVID-19 test. What technologies or systems of information transfer are used to send a test result, processed at a laboratory, back to health officials? Are these systems the same for both positive results and negative results? I wondered whether positive results might travel more quickly, due to the urgency they represent ([1] Council of State and Territorial Epidemiologists, 2019). I also wanted to know about the factors—say, geographic location of the testing site or proximity to a weekend or federal holiday—that might affect the variability of these timelines ([2] Appleby, 2020; [3] Sun & Waananen Jones, 2021).

Next, I wanted a deeper understanding of how the city’s COVID-19 death data has been classified. Is the city’s COVID-19 death toll built by examining cause-of-death records on death certificates, or is it compiled through disease surveillance data, built by tallying the deaths of known COVID-19 cases? Both of these approaches are sound epidemiologically, and both have been used by jurisdictions across the country, but they carry different advantages and disadvantages ([4] Nguyen & Schechtman, 2020). In a similar vein, I wondered whether the city’s COVID-19 deaths were published according to the date the death actually occurred, or by the date the death was reported to health officials ([5] Walker, 2020). I wanted to know whether and how probable COVID-19 deaths—deaths to account for the fact that access to testing has been limited since the start of the pandemic—are defined and included in Chicago’s official record ([6] Reinhard et al., 2020). I also sought to understand how Chicago officials have reflected on excess mortality, which is an assessment of deaths beyond normal measure that can help to build a more comprehensive view of a disaster’s toll ([7] Rosenbaum et al., 2021).

Finally, I set out to explore how the surrounding technical infrastructure might affect the data these pipelines produce. I approached this topic aware of the prolific role the fax machine plays in the production of public health statistics ([8] Downs et al., 2011; [9] Kliff & Sanger-Katz, 2020). I also knew that Chicago officials have not been shy about the maintenance needs of the technologies that facilitate public health data collection. In the city’s 2021 IT Strategic Plan, officials wrote frankly about the “significant challenges caused by paper-intensive systems, fragmented data and aging legacy systems” ([10] City of Chicago, n.d.). The report said: “Sub-optimal IT causes a significant drain on City resources as departments ‘make do’ with manual processes and staff struggle to maintain antiquated systems long past their utility” ([10] City of Chicago, n.d.). As demonstrated by jurisdictions like Oregon—where, in October 2021, state health officials disclosed that a “technical error” had caused an undercount of approximately 550 COVID-19 deaths ([11] Oregon Health Authority, 2021)—the infrastructures that facilitate public health data reporting need regular maintenance, quality checks, and care.

With these questions, I embarked on my SoReMo fellowship. Quickly, however, I realized that without more time, and especially without access to health officials at the city, it would be difficult for me to build the kind of comprehensive understanding I originally sought—one that would allow me to situate the specific traits of these data pipelines into a broader historical context. To that end, I shifted the framing of this project to focus instead on the information technology. Specifically, I set out to examine how public health data limitations have been communicated historically.

My research question then became this: Have Chicago’s public health data pipelines always been this fragile? If yes, I wanted to know if and how the public’s understanding of the data should somehow be altered. If no, I wondered whether we could then identify the specific policies and decisions that led to the current fragility. Given the breadth of scholarship on Chicago’s July 1995 heat wave, I decided to use this one event as a case study. I set out to examine how—and how well—death data was compiled during and immediately after the 1995 heat wave, and I began to explore what the history of the city’s public health data infrastructure might reveal about its form and function today.

It is important to state unequivocally that this project presumes the integrity of U.S. COVID-19 data. This data offers the best available insight into the ever-fluctuating state of the pandemic, and it can and should be used by policymakers, researchers, and journalists. As a growing body of research over the past two years has exposed, however, the information infrastructure that surrounds COVID-19 data is deeply fragile ([12] Arvisais-Anhalt et al., 2021; [13] Azzopardi-Muscat et al., 2021; [14] Badker et al., 2021; [15] Gardner et al., 2020). To work with this data responsibly, then, it is important to recognize, contextualize, and interpret the effects of this fragility. It is similarly important to understand that the data, while profoundly real, is not as precise as quantitative researchers might expect or want it to be.

Methodology

Once I identified the 1995 heat wave as a relevant case study, I targeted three primary avenues of research. First, I searched through academic literature, attempting to find studies that analyzed Chicago’s death toll from the 1995 heat wave. Next, I searched through government documents and reports about the heat wave. Finally, I turned to newspaper archives to examine how journalists during and immediately after the heat wave communicated data reporting delays. In each avenue, and, indeed, over the course of the whole project, Eric Klinenberg’s Heat Wave: A Social Autopsy of Disaster in Chicago provided foundational support to my research.

Historical Parallels and Perspective

Crisis strikes: Chicago’s July 1995 heat wave

In July 1995, a crushing heat wave hit the upper Midwest, with Chicago particularly experiencing extreme temperatures ([16] National Weather Service, 1995). Between July 12 and 16, temperatures in the city ranged from 93F to 104F, and the heat index spiked to 119F on July 13 ([17] Centers for Disease Control and Prevention, 1995). With adequate warning and proper precautions, heat-related deaths are preventable ([17] Centers for Disease Control and Prevention, 1995), and the National Weather Service had indeed issued timely advisories and warnings about the looming high temperatures ([16] National Weather Service, 1995).

Still, the heat wave was deadly; Chicago certified 465 heat-related deaths between July 11 and 27 ([17] Centers for Disease Control and Prevention, 1995). As was later revealed in an official disaster report produced by the [16] National Weather Service (1995), information warning about the potentially lethal nature of the heat “either failed to reach or was not used effectively by the people who could have prevented heat-related deaths” (p. viii). Even Chicago’s public health department had not been adequately prepared, with health officials expecting the heat wave to cause little more than discomfort ([16] National Weather Service, 1995). Undoubtedly, these misinformed expectations set the stage for what would soon become a fraught and very public matter of dispute about the heat wave’s death toll ([18] Klinenberg, 2015).

Edmund Donoghue, the Cook County medical examiner, noticed the lethality of the heat quickly, when his office began to report deaths in drastically higher numbers than previous weeks had seen ([18] Klinenberg, 2015). At the time, Cook County tended to report between 65 and 70 all-cause deaths each day, of which fewer than a third tended to fall under the medical examiner’s purview; the number of heat-related deaths that Donoghue’s office reported on July 15 alone was 162 ([16] National Weather Service, 1995). Indeed, on July 17, the New York Times published a front-page story declaring a preliminary death toll in the city of 118 but noted that Donoghue’s office was overwhelmed, with “some 300 bodies waiting in funeral homes to be examined for causes of death” ([19] Johnson, 1995).

As Donoghue’s staff was busy at work, the city’s morgue filled with bodies, hospitals quickly reached capacity, and emergency rooms began turning away ambulances ([16] National Weather Service, 1995). These problems were compounded by the social and structural barriers to care; the city’s fire department, for example, “had no centralized system for monitoring the number or the nature of the requests for service” ([18] Klinenberg, 2015, p. 133). Still, Chicago officials—including and especially then Mayor Richard M. Daley—refused to take responsibility for the catastrophe ([18] Klinenberg, 2015). The mayor’s administration downplayed the crisis, blaming the high and rising death toll on “residents who were already on the verge of death prior to the heat wave” ([20] Browning et al., 2006, p. 663). Indeed, the city’s early public relations strategy revolved predominantly and steadfastly around deflection and denial ([20] Browning et al., 2006; [21] Kass, 1995; [18] Klinenberg, 2015).

The challenges of “standardized and practical” criteria

Fueling the Daley administration’s denial was the fact that at the time, there was no existing standard definition for a heat-related death ([17] Centers for Disease Control and Prevention, 1995). Donoghue, the Cook County medical examiner, aptly recognized the dilemma this absence might cause, and he worked quickly to develop a definition to use as he examined every decedent ([16] National Weather Service, 1995).

According to Donoghue’s new criteria, a death would be classified as heat-related if it met the following conditions:

  1. A body temperature of 105F or above was measured at/shortly after the time of death; or
  2. Evidence of high environmental temperature in the place where the body was found was noted by rescuers; or
  3. The body was found in a decomposed state and was last seen alive at the time of the peak of the heat wave ([16] National Weather Service, 1995).

Unfortunately, Donoghue was responsible for examining only a small portion of deaths in the city; there were plenty of other people who completed official cause-of-death paperwork, and each did so according to their own best medical judgment—a sound practice epidemiologically ([22] National Center for Health Statistics, 2018) and also the reason why heat-related death tolls, when compiled from death certificate data, are widely regarded as undercounts ([23] Berko et al., 2014). Put another way, the mere existence of Donoghue’s guidelines did not translate into wide use. As the [16] National Weather Service’s disaster report (1995) stated, these classification discrepancies challenge heat wave death data comparisons of all kinds, between cities and across events. Indeed, standardized criteria like Donoghue’s has been imposed so infrequently that historical heat-related mortality data is generally rendered “irreconcilable” ([24] Rydman et al., 1999, p. 42).

Even if Donoghue’s conditions had been in place long before the heat wave, the real-time demands they required would almost certainly have deterred widespread adoption ([16] National Weather Service, 1995). In the field, first responders tried to work as quickly as possible to save lives, yet Donoghue’s criteria asked them to document precise context about the decedent’s surroundings and sensitive details about the decedent’s state. Donoghue’s criteria required information about the room temperature, the presence of air conditioning in the room, whether windows had been opened or closed, and whether the decedent appeared particularly decomposed ([18] Klinenberg, 2015). As the [16] National Weather Service (1995) report found: “One of the difficulties in measuring the mortality associated with a heat emergency is the lack of definition for heat-related death that is both standardized and practical” (p. 21). For these reasons, quantitative research on heat-related illness and death still faces classification challenges. As [25] Hartz et al. (2012) found, “There is no uniformity in heat-related morbidity data; types of data used, how data are reported, or in how they are analyzed” (p. 72).

In the aftermath of the heat wave, officials from the Centers for Disease Control and Prevention (CDC), outside medical examiners, and other heat experts studied the data and ultimately confirmed Donoghue’s numbers. As [18] Klinenberg (2015) noted, however, these official conclusions “never received the media attention accorded to Daley’s denials” (p. 173).

A lengthy process, in the process of modernization

During the COVID-19 pandemic, Americans turned regularly to state-controlled data dashboards, which monitored the flow of numbers about tests, cases, deaths, hospitalizations, and, eventually, vaccinations ([26] Mattern, 2021). Especially during periods when COVID-19 cases surged, many jurisdictions updated these dashboards every 24 hours. But though the dashboards often conveyed a genuine and urgent sense of real-time information transfer, the data itself quietly carried inherent delays ([27] Testa et al., 2020). For example, if a dashboard shows 50 new deaths added to a jurisdiction’s count since the previous day’s update, that almost certainly does not mean exactly 50 people died in the immediately preceding 24 hours. Rather, it simply means that health officials added 50 new deaths to the toll—deaths that probably but not definitely occurred somewhat recently ([5] Walker, 2020).

To a reasonable extent, death reporting lags are neither nefarious nor alarming; the process simply moves slowly and carefully to prioritize accuracy ([28] National Center for Health Statistics, 2021b). According to the National Center for Health Statistics (NCHS), the division of the CDC that produces public health statistics, not only does all-cause death reporting take time, but death reporting of COVID-19 deaths takes extra time. At the time of this writing, the agency’s Understanding the Numbers informational webpage reads: “While 80% of deaths are electronically processed and coded by NCHS within minutes, most deaths from COVID-19 must be coded by a person, which takes an average of 7 days” ([28] National Center for Health Statistics, 2021b).

To understand why this process moves slowly, it is important to understand what exactly death reporting entails. In the United States, the current version of the standardized death certificate facilitates the collection of three kinds of data: information about the manner and cause of death, information about the manner and place of disposition, and demographic information about the deceased ([29] Centers for Disease Control and Prevention, 2003). In many cases, it is a funeral director who oversees collection of this data ([30] Ventura, 2018). The funeral director works with the decedent’s family members or loved ones to gather demographic data. The funeral director also contacts the medical examiner, coroner, or certifying physician who signs off officially on the decedent’s cause-of-death information ([31] Brooks & Reed, 2015).

Facilitating collection of both demographic and cause-of-death information is not easy; as the NCHS has stated explicitly: “Bringing these two key sources together has long been a challenge, and in some cases, has seemed an insurmountable one” ([30] Ventura, 2018, p. 14). In the 1990s, the NCHS and the National Association for Public Health Statistics and Information Systems (NAPHSIS) began collective efforts to modernize parts of this process ([32] Westat, 2016), and by the early 2000s, the new Electronic Death Registration System (EDRS) started rolling out across the country ([33] National Center for Health Statistics, 2021a). EDRS technologies are state-managed systems intended to improve the timeliness and accuracy of death data:

The idea was for the systems to be Internet-based to speed up the processes for funeral directors and physicians to jointly record their data; this coincided with the maturing of the Internet in about 2000. Funeral directors have adopted the electronic systems very well and effectively. This has not been the case for physicians; at the time of this writing, more than 55% of death records are completely processed electronically. Most EDRS in the states, while not fully electronic, are nevertheless functioning and producing data. An EDRS that is not fully electronic is one that likely has the full (electronic) participation of funeral directors but not the participation of the physicians and other medical certifiers; the latter may continue to certify deaths manually. Because of these very different stakeholders involved in registering and certifying a death, with different training requirements and other challenges, fully functional EDRS take time to become established. ([30] Ventura, 2018, p. 14)

Across the country, the expanding rollout and growing widespread use of EDRS is working to improve the timeliness of mortality reporting ([34] Howland et al., 2015; [7] Rosenbaum et al., 2021). As [7] Rosenbaum et al. (2021) found, however, allocating funding to the maintenance and improvement of a state’s EDRS is critical to disaster planning. And funding is not the only concern. Maintaining an EDRS requires states to find and foster technical expertise, connect the web-based EDRS to other technologies already used in the jurisdiction, design the system such that regular use is encouraged, and provide options for funeral directors and medical certifiers in areas where internet access is limited ([32] Westat, 2016).

Discussion

What can Chicago’s 1995 heat wave death data teach us about how to interpret COVID-19 statistics? By identifying parallels between the two crises—from the limitations that complicated data collection ([16] National Weather Service, 1995) to the overlapping communities affected disproportionately ([35] Chatters et al., 2020)—we can place current challenges into perspective and see that data limitations are not new but rather the legacy artifacts of our country’s fractured public health information infrastructure.

Though modernization efforts, like the growing use of EDRS technologies, can expedite death reporting and help to mitigate the concerns of accuracy that come from manual data compilation ([34] Howland et al., 2015), death data reporting will always carry some delay ([28] National Center for Health Statistics, 2021b). It is therefore incumbent on quantitative researchers to approach mortality statistics with appropriate expectations about the inherent lack of real-time precision, and to recognize that this lack of real-time precision has often resulted in undercounts ([36] Stokes et al., 2021). To adjust for these known limitations, it is widely accepted to use statistical measures, like excess death counts, to attempt to capture truer death totals ([37] Rossen et al., 2020; [38] Rossen et al., 2021). Perhaps, though, through a data equity lens ([39] Jagadish et al., 2021), these measures deserve linguistic scrutiny for how they cast historically under-resourced populations as “excess.”

The colossal challenge then becomes how to communicate these limitations, both in scholarship and to the general public. Unfortunately, creators of data visualizations tend to shy away from attempting to represent uncertainty because there simply are no easy solutions ([40] Hullman, 2020). And though it might seem that more transparency and literacy around public health data will lead to increased trust, [41] Lee et al. (2021) found that over the course of the COVID-19 pandemic, people skeptical of government-imposed restrictions in fact embraced COVID-19 data, creating savvy “counter-visualizations” (p. 2) at prolific rates for the sake of producing data-driven arguments against widely accepted health protocols. Indeed, the researchers found that these anti-restriction data aficionados “often reveal themselves to be more sophisticated in their understanding of how scientific knowledge is socially constructed than their ideological adversaries, who espouse naive realism about the ‘objective’ truth of public health data” ([41] Lee et al., 2021, p. 2).

To protect against this kind of misuse of public health data, some might suggest health officials limit the amount of data they publish. This approach, however, comes with its own flaws; any reduction of public data will lead to legitimate questions about a lack of transparency.

A compromise might lie in the embrace of raw or dirty data. When deployed carefully, data known to be incomplete, inconsistent, tangential, or even error-ridden can in fact provide value by revealing meaningful context ([42] Gitelman, 2013). An examination of dirty data might lead to important clues about how and why certain metrics were collected in the first place ([43] Uprichard, 2011). For example, to better understand a jurisdiction’s capacity for compiling COVID-19 numbers, it might be worthwhile to examine how often its time-series data is revised and republished, and to analyze the scope of changes made between versions.

Perhaps no example of important context lurking within data is more apt than the history of disease classification itself. The International Classification of Diseases (ICD) is maintained by the World Health Organization, and it facilitates standardization for data about illness and mortality ([44] World Health Organization, 2021). In its original format, the ICD was introduced with 200 categories ([45] Bowker & Star, 1999). This was not because the list of possible diseases capped out at 200; rather, 200 “was the maximum number that would fit the large census sheets then in use” ([45] Bowker & Star, 1999, p. 46). This is an example of infrastructure imposing hidden constraints on the information it supports. As [45] Bowker and Star (1999) wrote: “Infrastructure does more than make work easier, faster, or, more efficient; it changes the very nature of what is understood by work” (p. 108).

The absence of data also deserves more attention. As we embark on the third year of the COVID-19 pandemic, researchers should interrogate the prevalence of mortality studies and recognize that deaths—classified as “excess” or not—might be a relatively easy metric to track, but deaths do not represent the full breadth of damage. As the pandemic carries on, a growing number of people are describing debilitating COVID-19 symptoms that linger ([46] Groff et al., 2021), but these experiences might never be represented in hospitalization or death metrics. The urgency of collecting data on long COVID is dire ([47] Michelen et al., 2021), but attempts to quantify its prevalence are challenged by a sweeping variety of symptoms, structural barriers to care, and disparate data collection methods.

Here, again, the historical record provides a grim parallel. Though the official death toll in the immediate aftermath of Chicago’s 1995 heat wave was tragic on its own, the authors of the National Weather Service report, produced in the aftermath of the crisis, warned that “the effects of a heat wave are often delayed,” with increases in illness and mortality for years and decades to come ([16] National Weather Service, 1995, p. 37).

So, instead of whittling away the dirt of data, carefully tidying it to be usable, or exploring only data that appears clean or whole enough, we might do well to heed [43] Uprichard’s (2011) argument. Researchers should expand their definition of valuable data and harness the possibilities inherent in data’s context. What might this look like in practice? Perhaps researchers should encourage government officials and media professionals to provide as much context about COVID-19 data as is practicable. This might mean pressing both government officials and the journalists who communicate health data for more information about what numbers do and do not mean, when and why numbers might change, and where and how metrics differ from those published by nearby jurisdictions.

Finally, as data researchers, we must do the work ourselves of recognizing this context. We must seek to appreciate and understand how individual data points flow from document to dashboard, and we must familiarize ourselves with the mechanics and limitations of the surrounding data ecosystems. As public health data becomes more politicized, an understanding of how these data systems function will serve as an important safeguard in future efforts to distinguish between infrastructural failure, mere incompetence, and malice.

Future Work

In more ways than one, this project lays a foundation for future research. There is no shortage of research to be done that examines how social stigmas, classification discrepancies, and technical flaws seep into health statistics—inadvertently or not, innocuously or not.

Continuing with a case study approach, important parallels can be made between the data compilation challenges experienced during the COVID-19 pandemic and those of other past crises. Historical efforts to identify, classify, and count early AIDS cases and deaths, for example, deserve rigorous and contextualized study. Though the process for measuring the prevalence of AIDS has become more sophisticated in recent decades, with this modernization, wholly new data compilation challenges have been introduced ([48] Brookmeyer, 2010). A deeply sensitive humanistic lens here is crucial.

Another example is found in maternal mortality data, which is complicated by the fact that each U.S. jurisdiction used its own timeline to adopt the latest version of the U.S. standardized death certificate ([49] MacDorman et al., 2016). Unlike previous versions, the new standardized death certificate does seek to collect data on maternal mortality, but the incongruous nature of the rollout complicates efforts to detect trends and compile national summaries ([29] Centers for Disease Control and Prevention, 2003).

Finally, looking ahead, it seems extremely prudent to foster a fuller understanding of the limitations of our death reporting systems, as a sweeping amount of research is urgently warning about an inevitable deluge of climate-related mortality ([50] Gosling et al., 2007; [51] McGeehin & Mirabelli, 2001; [52] Peng et al., 2011; [53] Petkova et al., 2014).

License

The author of this technical report, which was written as a deliverable for a SoReMo project, retains the copyright of the written material herein upon publication of this document in SoReMo Reports.

References

[1]
[2]
[3]
Sun, A. and Waananen Jones, L. (2021). The trouble with the case curve during the holidays. New York Times.
[4]
Nguyen, Q. P. and Schechtman, K. W. (2020). Confirmed and probable COVID-19 deaths, counted two ways. COVID Tracking Project.
[5]
Walker, P. (2020). Is there a right way to chart COVID-19 deaths over time? COVID Tracking Project.
[6]
Reinhard, B., Brown, E., Thebault, R. and Sun, L. H. (2020). CDC wants states to count “probable” coronavirus cases and deaths, but most aren’t doing it. Washington Post.
[7]
[8]
Downs, S. M., Anand, V., Sheley, M. and Grannis, S. J. (2011). The last mile: Using fax machines to exchange data between clinicians and public health. Online journal of public health informatics 3 ojphi.v3i3.3892.
[9]
Kliff, S. and Sanger-Katz, M. (2020). Bottleneck for u.s. Coronavirus response: The fax machine. New York Times.
[10]
City of Chicago. (n.d.). 2021 IT strategic plan.
[11]
[12]
Arvisais-Anhalt, S., Lehmann, C. U., Park, J. Y., Araj, E., Holcomb, M., Jamieson, A. R., McDonald, S., Medford, R. J., Perl, T. M., Toomay, S. M., Hughes, A. E., McPheeters, M. L. and Basit, M. (2021). What the coronavirus disease 2019 (COVID-19) pandemic has reinforced: The need for accurate data. Clinical Infectious Diseases: An Official Publication of the Infectious Diseases Society of America 72 920–3.
[13]
Azzopardi-Muscat, N., Kluge, H., Asma, S. and Novillo-Ortiz, D. (2021). A call to strengthen data in response to COVID-19 and beyond. Journal of the American Medical Informatics Association 28 638–9.
[14]
Badker, R., Miller, K., Pardee, C., Oppenheim, B., Stephenson, N., Ash, B., Philippsen, T., Ngoon, C., Savage, P., Lam, C. and Madhav, N. (2021). Challenges in reported COVID-19 data: Best practices and recommendations for future epidemics. BMJ Global Health 6 005542–2.
[15]
Gardner, L., Ratcliff, J., Dong, E. and Katz, A. (2020). A need for open public data standards and sharing in light of COVID-19. Lancet 21 E80.
[16]
National Weather Service. (1995). Natural disaster survey report: July 1995 heat wave. U.S. Department of Commerce, National Oceanic; Atmospheric Administration.
[17]
Centers for Disease Control and Prevention. (1995). Heat-related mortality — chicago, july 1995. Morbidity and Mortality Weekly Report 44.
[18]
Klinenberg, E. (2015). Heat wave: A social autopsy of disaster in chicago. University of Chicago Press.
[19]
[20]
Browning, C. R., Wallace, D., Feinberg, S. L. and Cagney, K. A. (2006). Neighborhood social processes, physical conditions, and disaster-related mortality: The case of the 1995 chicago heat wave. American Sociological Review 71 661–78.
[21]
Kass, J. (1995). Daley, aides try to deflect the heat. Chicago Tribune.
[22]
National Center for Health Statistics. (2018). Writing cause-of-death statements.
[23]
Berko, J., Ingram, D. D., Saha, S. and Parker, J. D. (2014). Deaths attributed to heat, cold, and other weather events in the united states, 2006-2010. National Health Statistics Reports 76 1–5.
[24]
Rydman, R. J., Rumoro, D. P., Silva, J. C., Hogan, T. M. and Kampe, L. M. (1999). The rate and risk of heat-related illness in hospital emergency departments during the 1995 chicago heat disaster. Journal of Medical Systems 23 41–56.
[25]
Hartz, D. A., Golden, J. S., Sister, C., Chuang, W. C. and Brazel, A. J. (2012). Climate and heat-related emergencies in chicago, illinois (2003-2006. International Journal of Biometeorology 56 71–83.
[26]
Mattern, S. (2021). A city is not a computer: Other urban intelligences. Princeton University Press.
[27]
Testa, C. C., Krieger, N., Chen, J. T. and Hanage, W. P. (2020). Visualizing the lagged connection between COVID-19 cases and deaths in the united states: An animation using per capita state-level data (january 22, 2020 – july 8, 2020). vol 19 The Harvard Center for Population; Development Studies.
[28]
National Center for Health Statistics. (2021). Understanding the numbers: Provisional death counts and COVID-19.
[29]
Centers for Disease Control and Prevention. (2003). U.s. Standard certificate of death.
[30]
Ventura, S. J. (2018). The u.s. National vital statistics system: Transitioning into the 21st century, 1990–2017. Vital and health statistics 62 1–84.
[31]
Brooks, E. G. and Reed, K. D. (2015). Principles and pitfalls: A guide to death certification. Clinical Medicine & Research 13 74–82.
[32]
Westat. (2016). Electronic death reporting system online reference manual: A resource guide for jurisdictions. National Center for Health Statistics, Division of Vital Statistics.
[33]
[34]
Howland, R. E., Li, W., Madsen, A. M., Wong, H., Das, T., Betancourt, F. M., Nicaj, L., Stayton, C., Matte, T. and Begier, E. M. (2015). Evaluating the use of an electronic death registration system for mortality surveillance during and after hurricane sandy: New york city, 2012. American Journal of Public Health 105 e55–62.
[35]
Chatters, L. M., Taylor, H. O. and Taylor, R. J. (2020). Older black americans during COVID-19: Race and age double jeopardy. Health Education & Behavior 47 855–60.
[36]
Stokes, A. C., Lundberg, D. J., Bor, J. and Bibbins-Domingo, K. (2021). Excess deaths during the COVID-19 pandemic: Implications for US death investigation systems. American Journal of Public Health 111 S53–4.
[37]
Rossen, L. M., Branum, A. M., Ahmad, F. B., Sutton, P. and Anderson, R. N. (2020). Excess deaths associated with COVID-19, by age and race and ethnicity - united states, january 26 - october 3, 2020. MMWR. Morbidity and Mortality Weekly Report 69 1522–7.
[38]
Rossen, L. M., Branum, A. M., Ahmad, F. B., Sutton, P. D. and Anderson, R. N. (2021). Notes from the field: Update on excess deaths associated with the COVID-19 pandemic - united states, january 26, 2020 - february 27, 2021. MMWR. Morbidity and mortality weekly report 70 570–1.
[39]
Jagadish, H. V., Stoyanovich, J. and Howe, B. (2021). COVID-19 brings data equity challenges to the fore. Digital Government: Research and Practice 2 1–7.
[40]
Hullman, J. (2020). Why authors don’t visualize uncertainty. IEEE Transactions on Visualization and Computer Graphics 26 130–9.
[41]
Lee, C., Yang, T., Inchoco, G. D., Jones, G. M. and Satyanarayan, A. (2021). Viral visualizations: How coronavirus skeptics use orthodox data practices to promote unorthodox science online. In Proceedings of the 2021 CHI conference on human factors in computing systems. CHI ’21 pp 1–8. ACM, Yokohama, Japan.
[42]
Gitelman, L. (2013). ’Raw data’ is an oxymoron. MIT Press.
[43]
Uprichard, E. (2011). Dirty data: Longitudinal classification systems. The Sociological Review 59 93–112.
[44]
[45]
Bowker, G. C. and Star, S. L. (1999). Sorting things out: Classification and its consequences. MIT Press.
[46]
Groff, D., Sun, A., Ssentongo, A. E., Ba, D. M., Parsons, N., Poudel, G. R., Lekoubou, A., Oh, J. S., Ericson, J. E., Ssentongo, P. and Chinchilli, V. M. (2021). Short-term and long-term rates of postacute sequelae of SARS-CoV-2 infection: A systematic review. JAMA network open 4 e2128568.
[47]
Michelen, M., Manoharan, L., Elkheir, N., Cheng, V., Dagens, A., Hastie, C., O’Hara, M., Suett, J., Dahmash, D., Bugaeva, P., Rigby, I., Munblit, D., Harriss, E., Burls, A., Foote, C., Scott, J., Carson, G., Olliaro, P., Sigfrid, L. and Stavropoulou, C. (2021). Characterising long COVID: A living systematic review. BMJ Global Health 6 e005427.
[48]
Brookmeyer, R. (2010). Measuring the HIV/AIDS epidemic: Approaches and challenges. Epidemiologic Reviews 32 26–37.
[49]
MacDorman, M. F., Declercq, E., Cabral, H. and Morton, C. (2016). Recent increases in the u.s. Maternal mortality rate: Disentangling trends from measurement issues. Obstetrics and Gynecology 128 447–55.
[50]
Gosling, S. N., McGregor, G. R. and Páldy, A. (2007). Climate change and heat-related mortality in six cities part 1: Model construction and validation. International Journal of Biometeorology 51 525–40.
[51]
McGeehin, M. A. and Mirabelli, M. (2001). The potential impacts of climate variability and change on temperature-related morbidity and mortality in the united states. Environmental Health Perspectives 109 Suppl 2 185–9.
[52]
Peng, R. D., Bobb, J. F., Tebaldi, C., McDaniel, L., Bell, M. L. and Dominici, F. (2011). Toward a quantitative estimate of future heat wave mortality under global climate change. Environmental Health Perspectives 119 701–6.
[53]
Petkova, E. P., Bader, D. A., Anderson, G. B., Horton, R. M., Knowlton, K. and Kinney, P. L. (2014). Heat-related mortality in a warming climate: Projections for 12 u.s. cities. International Journal of Environmental Research and Public Health 11 11371–83.