COVID19 update, April 18, 2020: community testing in Silicon Valley reveals huge “stealth infection numbers” — or does it?

A group of researchers around Eran Bendavid of Stanford (John Ioannides was among the many co-authors) yesterday released the results of their community testing effort in Santa Clara County, CA, as a preprint on MedRXiv: http://doi.org/10.1101/2020.04.14.20062463

Santa Clara County (where I was on assignment for a couple of years) is basically Silicon Valley: Palo Alto (of Stanford University fame), Mountain View (where Google’s campus is located), Sunnyvale, Cupertino (basically Apple City at this point), San Jose, and the like. As the authors explain:

At the time of this study, Santa Clara County had the largest number of confirmed cases of any county in Northern California (1,094). The county also had several of the earliest known cases of COVID-19 in the state – including one of the first presumed cases of community-acquired disease – making it an especially appropriate location to test a population-level sample for the presence of active and past infections.

So they recruited 3,439 volunteers through location-targeted ads on Facebook, and administered antibody tests to them. After discarding unusable results (unable to draw blood, volunteer appeared from outside county,…) this left them with 3,330 data points. The novel kit (Premier Biotech, Minneapolis, MN) they were using is not yet FDA-approved, so they ran calibration tests themselves at a Stanford lab, using positive and negative control samples [1].
Then they reweighted the results by sex (the sample skewed female), ethnicity, and ZIP code distribution to more closely match the overall Santa Clara County population. They also corrected for the test kit’s performance.

The results look like a bombshell, suggesting a Dunkelziffer (dark [case] number, stealth [case] number) as high as 50:1 or even 85:1.

The unadjusted prevalence of antibodies to SARS-CoV-2 in Santa Clara County was 1.5% (exact binomial 95CI 1.11-1.97%), and the population-weighted prevalence was 2.81% (95CI 2.24-3.37%). Under the three scenarios for test performance characteristics, the population prevalence of COVID-19 in Santa Clara ranged from 2.49% (95CI 1.80-3.17%) to 4.16% (2.58-5.70%). These prevalence estimates represent a range between 48,000 and 81,000 people infected in Santa Clara County by early April, 50- 85-fold more than the number of confirmed cases.

However (hat tip: Alex Pournelle, son of the late lamented Jerry Pournelle) biotech entrepreneur Balaji Srinivasan (himself a Stanford Ph.D.) posted an elaborate peer review on Medium in which he criticizes the statistical and sampling methodology of the authors. Let me summarize his two main critiques: (1) He points out that even the small number of false positives found in the manufacturer’s test calibration (none were found in Stanford’s own recalibration, but that is quite possible given the small sample) might mean a significant chunk of the detected rate could be false positives.
(2) In addition, as COVID19 tests were very hard to come by in Santa Clara County (or anywhere else) at the time, respondents to the ad would be self-selected for people suspecting they had COVID19 at some point, being unable to get tested for love or money, and jumping at the opportunity to get tested for free (my paraphrase).

While (1) would make me look toward the low end of the 95% confidence interval of the authors, the effect of (2) is hard to quantify. The authors would presumably retort that the two whole-community testing efforts known — Robbio, Italy (10%) and Gangelt, Germany (14%) — obtained even higher infection rates.

Erik Wingren drew my attention to an even more peculiar finding: in Boston, the denizens of a homeless shelter were tested [2]. “Of the 397 people tested, 146 people tested positive. Not a single one had any symptoms. […] The 146 people who tested positive were immediately moved to two different temporary isolation facilities in Boston. According to O’Connell, only one of those patients needed hospital care, and many continue to show no symptoms.” On the one hand, you’d expect this population to be extra vulnerable — on the other hand, some might argue that anybody who can survive on the streets for an extended period of time, exposed to every possible and impossible community pathogen, likely would have built up a pretty solid immune response. On the gripping hand [3], this is a completely novel pathogen. Any coronaviruses they would have been exposed to thus far would be a subset of common colds. (Most common colds are actually caused by rhinoviruses, which are a different family.)

The results of Eran Bendavid et al. appear to imply that the actual infection fatality rate (IFR) of COVID19 is closer to the 0.1% of a nasty seasonal flu — which of course would have drastic public policy consequences. Taking into account the criticisms of the open peer reviewer would revise the IFR upwards — but still nowhere near the 5-10% CFRs (case fatality rates) thrown around for some countries with problematic testing availability.
I would say that this 0.1% represents a lower bound for the IFR, and the about 1.15% IFR found in Israel (which counts asymptomatic positive cases as patients) with a “young” population pyramid, and the about 3.1% CFR (case fatality rate) with a much “older” population pyramid found in Germany, can be taken as upper bounds. (I note that the preprint of Bendavid et al. makes no mention of age distribution adjustment explicitly says “We did not account for age imbalance in our sample” — which is significant considering morbidity and especially mortality go up strongly with age.)
Iceland continues to test more people, at this point reaching 11% of its entire population (by far the largest of any country). They found 1,754 confirmed infections out of 39,536 tested, or 4.4% — but with a truly self-selecting sample (anybody who wants to do so can get tested for free in Iceland — but this sample would obviously be skewed toward people who suspect they may have been exposed). They have seen only 9 deaths — corresponding to an infection fatality rate of 0.5% that I suspect is fairly close to the true IFR for a typical European age pyramid.

FOOTNOTES:
[1] “Among 37 samples of known PCR-positive COVID-19 patients with positive IgG or IgM detected on a locally-developed ELISA test, 25 were kit-positive. A sample of 30 pre-COVID samples from hip surgery patients were also tested, and all 30 were negative. The manufacturer’s test characteristics relied on samples from clinically confirmed COVID-19 patients as positive gold standard and pre-COVID sera for negative gold standard. Among 75 samples of clinically confirmed COVID-19 patients with positive IgG [antibodies], 75 were kit-positive, and among 85 samples with positive IgM [antibodies], 78 were kit- positive. Among 371 pre-COVID samples, 369 were negative.”

[2] A discussion of homelessness in the USA, and its relation to the deinstitutionalization movement, would be fodder for another (perhaps future) blog post.

[3] I couldn’t resist the Niven/Pournelle reference

UPDATE: John Campbell in his video today mentions both the Stanford study and the Boston homeless shelter, and points out a Dutch dataset I wasn’t aware of. They checked blood of 10,000 regular blood donors for antibodies. 3% of all samples had COVID19 antibodies. Since John cannot see how blood donors would somehow be more susceptible for infection than the general population, he assumes the 3% figure is representative of the infection rate in the Dutch population. With a population of 17.3m, that implies 519,000 Dutch have been infected (most of them apparently asymptomatic or with mild symptoms they misattributed to common colds or flus) — about 3% of the Dutch population, surprisingly similar to the Stanford study, and about 17x more than the official 30,619 cases diagnosed. This also implies the actual IFR in the Netherland is not 3,459/30,619=11.2%, but 1/17th that=0.66%. Hmm… not far from the 0.5% or so from the Iceland data…