County-level CDC and DHS vaccination statistics for Wisconsin vary in important ways











The Federal Centers for Disease Control (CDC) and the Wisconsin Department of Health Services (DHS) both independently collect COVID-19 vaccination statistics. Their statistics for the state of Wisconsin vary. Statewide, the CDC counts 3,469,836 fully vaccinated individuals (as of 2021-12-01), while the DHS counts 3,268,304 (as of 2021-12-02). This gap of 201,532 is too large to be explained by small differences in the reporting period.

Proportionally, the differences are even larger at the county level. For instance, according to the CDC, 76% of Bayfield County residents are vaccinated, but the DHS places this figure at 66.4%. The opposite pattern is shown in neighboring Douglas County, where the CDC vaccination rate is 39.9%, and the DHS rate is 58.9%.

The DHS acknowledges differences between their data and the CDC, saying “Data on this page may differ from data reported on the CDC COVID Data Tracker due to the fact that data may be updated on different schedules and reflect data ‘as of’ different dates or times of day. There may also be a delay between the time a vaccination record appears in the the state system and when it is received by CDC.”1 Likewise, the CDC website states, “Data presented here use standard metrics across all United States counties. However, data on this page may differ from data on state and local websites. This can be due to differences in how data were reported or how the metrics are calculated. For the most accurate and up-to-date data for a specific county or state, visit the relevant state or lcal health department website.2 However, simply deferring to the DHS isn’t the best option in Wisconsin, because the CDC receives reports from some agencies which don’t share records with the state government.

There isn’t consensus about which set of numbers is best to use. The Milwaukee Journal Sentinel uses the DHS statistics in their vaccine tracker, while the New York Times uses the CDC numbers. As I dug into the details of each dataset is tabulated, I discovered that each has strengths and limitations. The two datasets need to be combined to create the most accurate picture of vaccinations at the county level in Wisconsin.

County level discrepancies

The map and table below show the differences between county-level CDC and DHS estimates downloaded on 2021-12-02.

In most counties, the CDC vaccination count is slightly higher than the DHS count. In a few counties, the CDC count is significantly higher. And in a handful of counties, the DHS count actually exceeds the CDC. All of these latter counties have something specific in common. They all border Minnesota.

I believe there are at least two reasons for this set of discrepancies. The CDC counts more vaccinations overall because they receive vaccination reports from more agencies than the DHS. However, the DHS does a better job of matching vaccine recipients to their home county. Each of these sources of error affect particular counties differently.

map

Difference between DHS & CDC vaccination rates

Difference between DHS & CDC vaccination rates

table

Data collection procedures

Calculating vaccination rates requires not just counting the number of vaccine courses distributed, but also figuring out where the receiving arms reside. People may receive their vaccine dose(s) in a different jurisdiction than the one they live in.

The CDC and DHS perform both steps differently. The CDC casts a wider net, collecting vaccine statistics from more providers than the DHS. But the DHS is better at figuring out the actual residence of vaccine recipients. These two limitations cause significant under-counts in different places.

The DHS collects its vaccine data from the Wisconsin Immunization Registry (WIR). After providing any vaccination (not just COVID), the medical provider updates that patient’s record in the WIR. The WIR also includes the patient’s current address. To calculate county-level vaccination rates, the DHS simply geocodes the address on file.3

However, the DHS clarified in an email that “veteran and tribal vaccinations aren’t reported to the state, but they are reported federally.”4

The CDC describes its process here. They receive data on doses administered from a wide range of sources, including typical providers like clinics and pharmacies but also FEMA, “dialysis centers participating in the Federal Dialysis Center Program, …. [and] federal entity facilities (e.g., Department of Defense, Veterans Health Administration, Indian Health Service, and Bureau of Prisons).” Perhaps the V.A. isn’t the only federal agency whose vaccinations aren’t reported to the DHS.

With this data, “CDC makes every effort to determine the county of residence for all administered and reported doses; however, . . . CDC has excluded from county-level summary measures vaccination records missing county of residence.”

If the CDC can’t match a vaccine recipient to their county of residence, they count it toward the state’s vaccination total, but don’t allocate it to a county. As of 2021-12-01, the CDC included 109,658 fully vaccinated inviduals in Wisconsin’s total which they couldn’t match to any particular county. The DHS performs much better on this metric. They left only 22,887 vaccinated individuals unassigned.

Sources of undercount

The DHS doesn’t count vaccinations by tribal governments, while the CDC does. This explains large under-counts in the DHS data for Bayfield, Forest, and Shawano counties. Bayfield County is home to the Red Cliff Band, who conducted their own vaccination campaign. Ten percent of Bayfield County identified as Native American in the 2020 Census. The CDC’s vaccination rate is 10 percentage points higher than that of DHS.

Likewise, 8% of Shawano County (home to the Stockbridge-Munsee nation) is Native American. The CDC’s vaccination rate is 6 points higher than that of DHS. And the CDC rate is 9 percentage points higher in Forest County, where two tribal governments are located and 13% of the population is Native American.

In most counties, the CDC counts only slightly more fully vaccinated residents than the DHS. I suspect these smaller differences are due to vaccines distributed by other federal agencies (like the V.A.) and only counted by the CDC.

What about Douglas and the other counties bordering Minnesota where the DHS counts a much higher number of vaccine recipients than the CDC? Here is where the Wisconsin Immunization Registry (WIR) shines. WIR has a reciprocal data sharing agreement with the state of Minnesota.5 Recall that the DHS counts 58.9% of Douglas county’s population as fully vaccinated, while the CDC thinks only 39.9% is. It’s entirely believable to me that 19% of Superior, Wisconsin’s population gets medical care in their much larger neighbor of Duluth, Minnesota. A similar pattern likely repeats itself in the Wisconsin exurbs of the Twin Cities.

Thanks to the data-sharing agreement, all these Wisconsinites getting vaccinated in Minnesota have their medical records properly updated in WIR, and consequently correctly counted by the DHS. Because the CDC counts so many fewer vaccinated individuals in these counties, I suspect they are failing to identify the Wisconsin county of residence for those receiving vaccines in Minnesota. I don’t know if these individuals are counted toward the state totals for Minnesota or Wisconsin.

I remain confused by a couple of counties where the CDC counts significantly more vaccinated individuals, but there isn’t a tribal government or large Native American population to explain the gap. Most notably, these include Rock, Florence, and Grant Counties. Perhaps there are large federal vaccination programs in these counties I’m not aware of.

Implications for accurate county statistics

Ideally, either the DHS would start collecting statistics from the tribes and federal agencies. But since the WIR has been around for 20 years, that kind of change seems unlikely. More plausibly, the CDC could change their methodology to incorporate the address-matching already being done by the WIR.

Both the CDC and DHS statistics have geographically-specific issues. Because the DHS doesn’t include tribal vaccinations, it at least seems best to use the CDC’s numbers for Bayfield, Forest, and Shawano counties. On the other hand, the pattern of counties on Minnesota’s border being undercounted in the CDC data seems too obvious to ignore.

At minimum, I recommend using CDC data for the three counties mentioned above and DHS data for counties where the DHS total exceeds that of the CDC. For the remaining counties, I think the CDC is a better source as it incorporates vaccines distributed by federal agencies.

In reality, this kind of ad hoc adjustment isn’t practical for daily trackers like those maintained by the Milwaukee Journal Sentinel or the New York Times. The MJS makes good use of the more granular geographies (tract zip code) provided by the DHS but not the CDC. The NYT tracker covers the entire country, so they need a consistent, national dataset. They are unlikely to incorporate data from individual states using manual judgment calls. Still, analysts studying and modeling vaccination rates at the county level would likely benefit from considering these data accuracy improvements.

Another way to improve the county-level dataset is to use 2020 census counts as the denominator for calculating total-population vaccination rates. Currently, the CDC and DHS both use the Census Bureau’s 2019 population estimate. This is because the Census Bureau hasn’t yet released detailed age tables from the 2020 census, and the DHS/CDC need consistent population counts across their various age-specific statistics.

Most of the differences between the 2019 estimate and the 2020 count are small, but some of them are substantial. Sawyer and Door Counties grew by 9%, Bayfield and Iron by 8%. Menominee County shrank by 7%.

Adding all these steps together makes changes that matter. It must concern Douglas County greatly whether they are among the worst vaccine laggards or are in fact better than average. When we substitute the more recent census count into the denominator, suddenly Menominee County has the highest vaccination rate in the state at 76.1%. Wisconsin’s smallest, poorest, and least white county–essentially coterminous with the Menominee Nation–has a higher vaccination rate than even Dane County. That is a significant fact, but it’s currently obscured by less accurate statistics.

Comparison of total population vaccination rates from the CDC and WI DHS, based on either the 2019 population estimate or the 2020 census count


 

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.