Measurement uncertainty

Zechariah 2:2

This post is quite dry, so I smurfed it up a bit.

The whole business of tracking Covid-related figures is getting fuzzier and fuzzier. Let me explain using ICU occupancy in Germany as an example.

DIVI (Deutsche Interdisziplinäre Vereinigung für Intensiv- und Notfallmedizin, a name as German as can be) publishes daily counts of Covid patients on ICU. As far as I can see, there is no distinction at all between the “fors” and the “withs”. For calendar weeks 39-42 of 2022, a sum total of 5,921 patients has been recorded. Hospitals are also required to track the patients’ vaccination status. However, these data are not published directly, but only weekly (and from now on only monthly, unfortunately) after processing by the RKI (Robert-Koch-Institut). Of the 5,921 patients, vaccination status is reported to be known for 3,258 – quite underwhelming for a number resulting from a mandatory reporting in a rich first-world country, if you ask me. These 3,258 are stratified by vaccination status as follows: 2,880 (88.4%) had at least one jab, 2,785 (85.5%) had at least two jabs, 2,499 (76.7%) had at least three, and 647 (19.9%) were members of the elite four-jabs club.

Of course, some words might be spent on multinomial confidence intervals or such beasts, but the uncertainty is due to the absolute numbers being so small (in a country of 84 million people), and we should be happy that they are that small. In my very first substack post, almost a year ago, I presented the analogy to credit risk modelling: parameterization of a credit risk model (a rating model, or a credit portfolio model) is much less plagued by estimation error if you have observed lots of defaults in the past, but of course the very fact that your bank (i.e., employer) is still in business means that defaults have not been too frequent.

On the other hand, numbers seem to be getting reassuringly large when we try to compare ICU vaccination rates to population vaccination rates. For the time period considered, population-level rates reported are 77.9% (1+ jabs), 76.3% (2+ jabs), 62.3% (3+ jabs), and 11.8% (4+ jabs). All these proportions are lower than those reported for ICU patients. Does this mean that vaccination actually increases your chances of ending up in ICU with or for Covid? Not exactly. The age distribution of ICU patients is very different from the age distribution of the population, and this has to be taken into account. Unfortunately, early on in the pandemic, authorities in Germany decided only to count vaccinations in four age categories (0-11, 12-17, 18-59, 60 and above), and there is no chance of improving upon this now. According to DIVI, young patients (0-17) made up around 1.9% of the 5,921 ICU patients, middle-aged patients (18-59) accounted for 15.8%, and 82.3% of patients were 60 or older. We have no idea if the same proportions apply to the sub-sample of 3,258 patients with known vaccination status, but they will have to do. Based on the vaccination rates reported for the different age categories, if vaccination status was independent from ICU admission, we would expect the following baseline vaccination rates among ICU patients: 89.0% for 1+ jabs, 88.0% for 2+ jabs, 81.0% for 3+ jabs, and 26.8% for 4+ jabs. There seems to be a modicum of vaccine efficacy left, but the benefits hardly outweigh the costs (horrifying side effects, authoritarianism, societal disruption).

And even the numbers just computed are not set in stone, but subject to considerable uncertainty. To see this, let us concentrate on the 60+ age category, which dominates the numbers anyway. Vaccination rates reported for this age category are 90.8% (1+ jabs), 90.0% (2+ jabs), 85.0% (3+ jabs), and 31.7% (4+ jabs). How do you arrive at these figures? You start with a population figure, you count jabs, and then you divide the latter by the former. The problem is that populations change. There are around 24 million people in the 60+ category in Germany. Each year, around 0.9 million of them (or around 4%) die, and are replaced by roughly the same number of 59-year olds. The true vaccination rate depends on the vaccination status both of the deceased and of the newcomers. The extremes are as follows:

• Only vaccinated people die, and all the 59-year old newcomers are unvaccinated.

• Only unvaccinated people die, and all the 59-year old newcomers are vaccinated.

After one year, therefore, in the first case the true vaccination rate is 4 percentage points lower than reported, and in the second case it is 4 percentage points higher. Of course, both these extremes are unrealistic, but nevertheless useful for quantifying the uncertainty. Note again that this uncertainty – in contrast to the one discussed above, regarding the small number of ICU patients – is not due to sample size but to the passage of time, and it will only get worse. At a certain point, the RKI’s reporting of vaccination rates will have become completely useless, and true vaccination rates would have to be estimated by re-sampling of the population. But such random sampling did not happen throughout the pandemic (although I can not have been the only one to propose it), and it never will.