Data collection procedures

Calculating vaccination rates requires not just counting the number of vaccine courses distributed, but also figuring out where the receiving arms reside. People may receive their vaccine dose(s) in a different jurisdiction than the one they live in.

The CDC and DHS perform both steps differently. The CDC casts a wider net, collecting vaccine statistics from more providers than the DHS. But the DHS is better at figuring out the actual residence of vaccine recipients. These two limitations cause significant under-counts in different places.

The DHS collects its vaccine data from the Wisconsin Immunization Registry (WIR). After providing any vaccination (not just COVID), the medical provider updates that patient’s record in the WIR. The WIR also includes the patient’s current address. To calculate county-level vaccination rates, the DHS simply geocodes the address on file.³

However, the DHS clarified in an email that “veteran and tribal vaccinations aren’t reported to the state, but they are reported federally.”⁴

The CDC describes its process here. They receive data on doses administered from a wide range of sources, including typical providers like clinics and pharmacies but also FEMA, “dialysis centers participating in the Federal Dialysis Center Program, …. [and] federal entity facilities (e.g., Department of Defense, Veterans Health Administration, Indian Health Service, and Bureau of Prisons).” Perhaps the V.A. isn’t the only federal agency whose vaccinations aren’t reported to the DHS.

With this data, “CDC makes every effort to determine the county of residence for all administered and reported doses; however, . . . CDC has excluded from county-level summary measures vaccination records missing county of residence.”

If the CDC can’t match a vaccine recipient to their county of residence, they count it toward the state’s vaccination total, but don’t allocate it to a county. As of 2021-12-01, the CDC included 109,658 fully vaccinated inviduals in Wisconsin’s total which they couldn’t match to any particular county. The DHS performs much better on this metric. They left only 22,887 vaccinated individuals unassigned.

Sources of undercount

The DHS doesn’t count vaccinations by tribal governments, while the CDC does. This explains large under-counts in the DHS data for Bayfield, Forest, and Shawano counties. Bayfield County is home to the Red Cliff Band, who conducted their own vaccination campaign. Ten percent of Bayfield County identified as Native American in the 2020 Census. The CDC’s vaccination rate is 10 percentage points higher than that of DHS.

Likewise, 8% of Shawano County (home to the Stockbridge-Munsee nation) is Native American. The CDC’s vaccination rate is 6 points higher than that of DHS. And the CDC rate is 9 percentage points higher in Forest County, where two tribal governments are located and 13% of the population is Native American.

In most counties, the CDC counts only slightly more fully vaccinated residents than the DHS. I suspect these smaller differences are due to vaccines distributed by other federal agencies (like the V.A.) and only counted by the CDC.

What about Douglas and the other counties bordering Minnesota where the DHS counts a much higher number of vaccine recipients than the CDC? Here is where the Wisconsin Immunization Registry (WIR) shines. WIR has a reciprocal data sharing agreement with the state of Minnesota.⁵ Recall that the DHS counts 58.9% of Douglas county’s population as fully vaccinated, while the CDC thinks only 39.9% is. It’s entirely believable to me that 19% of Superior, Wisconsin’s population gets medical care in their much larger neighbor of Duluth, Minnesota. A similar pattern likely repeats itself in the Wisconsin exurbs of the Twin Cities.

Thanks to the data-sharing agreement, all these Wisconsinites getting vaccinated in Minnesota have their medical records properly updated in WIR, and consequently correctly counted by the DHS. Because the CDC counts so many fewer vaccinated individuals in these counties, I suspect they are failing to identify the Wisconsin county of residence for those receiving vaccines in Minnesota. I don’t know if these individuals are counted toward the state totals for Minnesota or Wisconsin.

I remain confused by a couple of counties where the CDC counts significantly more vaccinated individuals, but there isn’t a tribal government or large Native American population to explain the gap. Most notably, these include Rock, Florence, and Grant Counties. Perhaps there are large federal vaccination programs in these counties I’m not aware of.

Implications for accurate county statistics

Ideally, either the DHS would start collecting statistics from the tribes and federal agencies. But since the WIR has been around for 20 years, that kind of change seems unlikely. More plausibly, the CDC could change their methodology to incorporate the address-matching already being done by the WIR.

Both the CDC and DHS statistics have geographically-specific issues. Because the DHS doesn’t include tribal vaccinations, it at least seems best to use the CDC’s numbers for Bayfield, Forest, and Shawano counties. On the other hand, the pattern of counties on Minnesota’s border being undercounted in the CDC data seems too obvious to ignore.

At minimum, I recommend using CDC data for the three counties mentioned above and DHS data for counties where the DHS total exceeds that of the CDC. For the remaining counties, I think the CDC is a better source as it incorporates vaccines distributed by federal agencies.

In reality, this kind of ad hoc adjustment isn’t practical for daily trackers like those maintained by the Milwaukee Journal Sentinel or the New York Times. The MJS makes good use of the more granular geographies (tract zip code) provided by the DHS but not the CDC. The NYT tracker covers the entire country, so they need a consistent, national dataset. They are unlikely to incorporate data from individual states using manual judgment calls. Still, analysts studying and modeling vaccination rates at the county level would likely benefit from considering these data accuracy improvements.

Another way to improve the county-level dataset is to use 2020 census counts as the denominator for calculating total-population vaccination rates. Currently, the CDC and DHS both use the Census Bureau’s 2019 population estimate. This is because the Census Bureau hasn’t yet released detailed age tables from the 2020 census, and the DHS/CDC need consistent population counts across their various age-specific statistics.

Most of the differences between the 2019 estimate and the 2020 count are small, but some of them are substantial. Sawyer and Door Counties grew by 9%, Bayfield and Iron by 8%. Menominee County shrank by 7%.

Adding all these steps together makes changes that matter. It must concern Douglas County greatly whether they are among the worst vaccine laggards or are in fact better than average. When we substitute the more recent census count into the denominator, suddenly Menominee County has the highest vaccination rate in the state at 76.1%. Wisconsin’s smallest, poorest, and least white county–essentially coterminous with the Menominee Nation–has a higher vaccination rate than even Dane County. That is a significant fact, but it’s currently obscured by less accurate statistics.