I’ve written a longish article called ‘The Real Problem With Epidemiological Models’, which Toby Young has published at his Lockdown Sceptics site. You can read it here. (This link won’t be clickable from the front page of Hector Drummond Magazine, you need to click on this post first, then the link will be clickable.)
Share this article on social media:
35 thoughts on “The Real Problem With Epidemiological Models”
That’s a useful post. It highlights that the main problem isn’t the models – it’s the theory that underlies them – how viruses work doesn’t seem understood nearly well enough to make reliable predictions about their behaviour and evolution.
At best, as you say, it might be possible to extract numbers for R0 and IFR and variability from historical data. But these are averages over time, location and people. Although Ferguson’s model includes variation of R0 between locations and people, it doesn’t allow for R0 to change “spontaneously” over time. This isn’t surprising as it would have needed a forecast for how the virus would evolve.
It may be that many of the failures of prediction are because of the assumption that the virus has a consistent R0 while the infected population gets infected, builds up immunity and so on. But the virus is also changing, in different ways and rates in different places.
Maybe what’s being seen in the fairly common shape of the case and mortality curves is the virus moving to a lower R0 (and possibly IFR) as part of its natural evolution. C-19’s initial R0 led to the hosts it “wanted” to infect cutting their links and stopping C-19 from spreading. C-19 might respond by lowering its R0, so the hosts restore their links and C-19 can spread again although more slowly than before. Perhaps that’s how humans and flu “learned” to co-exist.
I must apologise in advance to all those people who have provided graphs and numbers and stats=good on you all for trying so hard to provide useful information, but I’m now utterly sick of the foregoing, and I’m afraid I now lump you in with all the government experts and others who produce what to me are now a lot of esoteric figures that have ceased to have any meaning for ordinary people like me. Guesses, estimates, projections, stabs in the dark? I don’t care!
I’m sorry to say that I am now reduced to being someone who no longer cares how many have died, and where and how those unfortunates have met their end. I’m sick of numbers, I’m sick of politicians trying to talk their way out of a sticky situation. I’m just sick of all of it.
Again, apologies to all those who’ve provided the stats, but I’m now retiring!
Good luck to all.
“It may be that many of the failures of prediction are because of the assumption that the virus has a consistent R0 while the infected population gets infected, builds up immunity and so on.”
Isn’t the failure of prediction due to a more fundamental reason that, particularly in the early stages of the epidemic, there’s too many unknowns to predict with any accuracy? In other words, are we kidding ourselves that this kind of prediction is actually possible?
I think I too have got to that stage John: those doing the data can be shown to be correct; but one cannot persuade the scared;
I too am exhausted and worn out at the madness.
“In other words, are we kidding ourselves that this kind of prediction is actually possible?”
absolutely Dene;
Thirded.
@John Wilkinson
+1 Although I never cared and have a “keep calm and carry on” attitude. If I’m infected so be it, when I die so be it
You may like these
From 27 Feb 2020, informative
Where Do New Viruses Come From?
https://www.youtube.com/watch?v=NJLXdsO1GBI
The Two Viruses That We’ve Had For Millions of Years – and how it may explain Wuhan CV19 Virus
Viruses have been infecting Homo-erectus for millions of years and we’re still here and thriving
https://www.youtube.com/watch?v=NHTniCvTLDY
So we hear that supposedly 17% of Londoners tested antibody positive, and just 5% of those outside the capital. But there’s surely a lot of small print behind those numbers. Firstly, the test sample was done in late April/early May. Secondly, antibodies don’t show up reliably for a fortnight after infection, or about ten days after symptoms. So that implies we are looking at cases at least ten days prior to sampling. That would imply we should adjust the infection rates for the present situation. London had already seen over 80% of its cases to date by that time, so the present prevalence level of having been infected is likely to be of the order of 21%. But outside, we are probably looking at something like double the measured prevalence.
Prediction is certainly possible; the issue is whether it’s accurate enough to be useful. In the case of a virus, if it has an R0 value of above ~1.5 and infected people require significant NHS resources, if it’s allowed to run unchecked, the NHS would be unable to cope by a large margin. AFAIK, no one has so far suggested that in March C-19 had an R0 of below 1.5: the central value at that time was ~2.4. Therefore the prediction made by Ferguson et al in mid-March was quite accurate enough to predict with confidence that unless significant action was taken the NHS would be overwhelmed, not by a small amount by a factor of more than 30.
It was an “easy” prediction in that it’s very hard for it to have been wrong.
The interesting question is what’s happened to R0 since then? Is the fairly common shape of infection/mortality curves due to similar international lockdowns or R0 changing as the virus evolves?
If models worked every advanced country would have one to take into account its own population, geography, climate etc. That this not the case means either Ferguson is a giant astride epidemiology, or everyone else except us knows they don’t work.
Thanks to those who either commiserate or understand my semi-rant and to Pcar for the info-much appreciated.
Again, no offence meant towards those who do these graphs etc, good luck to you.
I just want out of the utterly ridiculous situation to which our country has now descended.
‘It was an “easy” prediction in that it’s very hard for it to have been wrong.’
Are you saying that it was right and we would have had 500,000 deaths without lockdown, or that the prediction was simply in line with the limited data we had at the time?
“All models are wrong, but some are useful”. George Box
JW – ditto.
Neither of your conclusions is true. You don’t need the details of F’s model to arrive at F’s conclusions. They’re so strong that the simplest models give much the same results. F’s model is a diversion. No giants needed and F’s model, at least at the level of nationwide predictions, is “fit for purpose”.
With the exception of Sweden (which with a combination of rules and voluntary measures reached much the same state as other countries) all major countries locked down in a similar way to that which F’s paper recommended because they’d all reached similar conclusions as to what was otherwise likely to happen,
“Therefore the prediction made by Ferguson et al in mid-March was quite accurate enough to predict with confidence that unless significant action was taken the NHS would be overwhelmed, not by a small amount by a factor of more than 30.”
I do wonder what you mean by accurate? All the science I’ve seen on these models is very clear, they are incapable of predicting anything. There primary role is propaganda. I see zero evidence the model predicted anything any more accurately than puling numbers out of the air.
Japan isn’t a major country?
South Korea isn’t a major country?
Australia isn’t a major country?
Taiwan isn’t a major country?
R is a theory. An attempt to explain a virus’s behavior. The model is a major problem because it is creating an impression of ”science’ that predicts. Ferguson has been pushing this for many years, he just wasn’t listened too much. This time everyone has used exactly the same tool.
Increasingly I think there is a strong possibility the models have actually created the deaths. Covid is not a clean killer, it’s hazy as to it being responsible for a death. Creating the impression it will kill millions creates confirmation bias. So, just how many of these deaths would have been noticed if we did not have Covid! on everyone’s minds? Am I alone in thinking this is a major contributor to all of this?
I’d like to see the model run on Japan and explain exactly what has happened there.
The latter: given the inputs (essentially R0 and IFR) the conclusions – ~500,000 deaths and a collapsed NHS – were inevitable. The inputs were of course uncertain but there were no more reliable estimates available. I’m not sure that situation has changed very much in the meantime. Estimates for both R0 and IFR still vary widely, complicated by political, cultural, environmental and demographic issues. Also lockdowns have changed R0 so its “true” value can’t be observed.
On accuracy, Ferguson didn’t have to hit a treble twenty. He didn’t even have to hit the dartboard. Anywhere on the wall would have done.
Given R0 of ~2.4 and IFR of ~1%, more than 80% of the population would be infected and more than 500,000 people would die. That’s really what those values of R0 and IFR mean and the details of F’s model (or the 20% or 30% uncertainties or errors which people say they’ve found) are just fine tuning.
If 400,000 deaths were forecast rather than 500,000 and the NHS’s capacity was exceeded only 25 times over rather 30+, it really wouldn’t make a material difference to the conclusion.
On Japan – me too. And on every other country. But I’m fairly sure it will do quite well. The piece I did on this website a while ago (“Why the Swedish model was wrong” or something of the kind) involved understanding why a model, based on F’s model, had produced such bad predictions. One obvious factor was that they’d used a value for R0 based on data which gave a misleadingly high value when more relevant data would have reduced their predictions considerably.
Unlike the Swedes, F didn’t have a better source of data relevant to the UK (the case data was hopeless at that time and couldn’t be relied on) so he had to use estimates from elsewhere. Maybe, just as in the Swedish case, the local value of R0 was significantly lower than the estimate. F can’t be blamed for the lack of useful data – government/PHE/NHS will doubtless blame each other for that. He did what he could with what he had.
I think he could and should have stressed the uncertainty of his predictions but, without counterbalance from e.g. economists on the costs of LD, the argument against LD would still have been very weak.
What was really missing was any equivalent of F’s work on the costs of LD. AFAICT, the LD process was originally set down about 14 years ago, argued about a little and then became a fairly settled policy but without anyone properly assessing the huge economic and social costs.
So F and colleagues presented the case for LD from the epidemiological p-o-v and there was no case to be made against it. Even had people wanted to argue that way, they had no backing material – reports, papers, analysis, conferences, authorities – to support their argument.
Different answers in different cases:
Australia and S Korea had LDs of a similar severity to the US (see Peter Forsythe’s post on this site).
Taiwan had prepared a system and technology for this situation and so didn’t need a full-scale LD. Had other countries done likewise, they might also have avoided LD.
Japan seems to be an anomaly but may not be. They closed schools and “urged” people to socially distance, businesses to close and people to stay at home except for essential requirements. The authorities relied on peer pressure and it seems to have worked in that many businesses did close, people stopped using public transport and so on. They reached de facto LD similar to elsewhere but voluntarily.
As well as the power of peer pressure, another aspect of Japanese culture is personal hygiene, handwashing and cleanliness. This helped stop the spread of the virus in the pre-voluntary-LD days which was likely a big factor in the much larger number of infections and deaths in other countries.
“Australia and S Korea had LDs of a similar severity to the US (see Peter Forsythe’s post on this site).”
I’mm actually in Australia. It’s nowhere as strict as that in the UK. Most businesses remained open, retail, hairdresses, coffee shops etc. Restrictions on ‘non essential’ travel, enforced to a widely different level depending on state and proximity to police.
As far as I can see there is no evidence the different approaches have made any difference to the outcomes.
Watch this Clip from 5mins 37secs in, is very interesting relating to the number of people who’ve had the virus https://youtu.be/DKh6kJ-RSMI
As far as im concerned its clear to see this is whats happenened and what has happened in other countries aswell, the virus has already been exposed to nearly the whole population especially in places like london. Ten’s of thousands of tests carries out last weekend in the capital and single figure positives. There is no Way people are adhereing that well to the lockdown for this to be the reason. 50% of the city are still travelling to work and many in jobs where social distancing is impossible travelling on far from empty public transport.
Sad confrimation that all of this has been a huge waste of time and many have already paid in deaths caused by the lock down and I fear many more will pay in quality of life for years to come.
I truly struggle to believe, in a decision of that magnitude, there was no effort made to understand the impact of the proposed policy. You do not need any backing materials to understand shutting down an economy will cost lives, and plenty of them.
Economic models exist just as much as epidemically ones. Children could do better.
I have always suspected this virus was widespread very early. If it was anywhere near as infectious as claimed it should have spread worldwide long before the beginning of the year. When NZ locked down I was amazed to see there had been no random testing to see just how widespread it was. Everywhere seems to be the same, testing was limited to those with symptoms. Like any iceberg, what matters in what is under the water.
I found this on twitter, some indication of this;
https://pbs.twimg.com/media/EX6NlhtVcAIsJjQ?format=jpg&name=4096×4096
There’s no off-the-shelf economic model to plausibly cover lockdown. People are still arguing about what caused and ended the Great Depression. This is an entirely unprecedented situation in which no one has a clear idea of what will happen next week, let alone what the situation will be in a year’s time. And macroeconomics has a dismal record of successful prediction even in better times.
In contrast, epidemiologists have a solid theory with a good track record in many cases. People pick out the perceived failures and somehow forget or are unaware of the reliable calculations of herd immunity which support immunisation programmes for, say, measles, mumps, rubella, polio, hepatitis.
Set against a prediction of half-a-million deaths in two months from people with a track record of successfully helping to control what had been very dangerous infections, anyone vaguely forecasting economic doom wouldn’t have stood a chance.
I have great empathy for John W.’s frustration. We know that C-19 is not unusually infectious; we know that it is not unusually fatal; and we know that Western politicians have dutifully followed the Chinese lead in imposing Lock Downs which are probably responsible for as much medical harm as medical benefit. Now, perhaps, it is time to “Move On”, as the Clinton groupies said when the facts of Bill’s philandering became unavoidably clear.
Move On — to what is still to come: the economic fallout from the inappropriate Lock Downs.
Western country economies are heavily financialized — lots of bankers, lawyers, stock analysts, intermediaries getting rich pushing pieces of paper around, and hiring other people to provide real services like restaurants and haircuts. In contrast, China’s economy is heavily focused on the production of real goods — iPhones, laptops, automobiles, ships, nuts, bolts, and much of everything else one can buy in a Western store.
What happens when the Lock Downs fade away? The damage to financialized Western economies is going to be much larger than to the productive Chinese economy. What will be the consequences of that? How can we minimize any negative effects from those unavoidable consequences?
‘It was an “easy” prediction in that it’s very hard for it to have been wrong. ‘
If the prediction was based on 80% of the population becoming infected, and the IFR applying equally to all infections, was that a plausible scenario? I’ll do some digging because it’ll be interesting to see if a coronavirus is likely to have such penetration into the population and remain equally lethal.
The IFR is an average over all demographics, not equal for all infections.
I actually wrote that the Australian LD was similar to the US rather than UK (according to the analysis quoted in Peter Forsythe’s post). I haven’t looked into the detail beyond that as I’m not sure that comparison of northern and southern hemisphere countries is very useful until the seasonal behaviour of the virus is better understood.
The US is shutdown to quite a large degree in various places.
Totally agree about southern hemisphere. It is still summer and the test is to come over winter. I do, however, wonder if it’s not that the virus has already been through. That would start to explain the relative performance of Pacific rim nations and Europe/New York. The geographic differences are very odd to me.
Simon Anthony
You’re missing my point. No country said “our model predicts a peak of x deaths Tuesday week and we’re going with that.” Anyone can look at an exponentially rising death rate and predict that half country will die if it isn’t stopped. So, as I said, nobody had a model that worked because models don’t work. The UK stupidly believed they had a model that did work.
I gave up expecting a reply to my comment on your article, but as mentioned above and in my comments, any projection, whether using a sophisticated (but likely inaccurate model) or simple logarithms showed that healthcare would be overwhelmed within a couple of weeks.
Immediately before lockdown, Cases were doubling every three days and deaths were doubling every two days (which you refute, despite the obvious calculation). Faced with such a trajectory, action to reduce contacts was the only available action.
Why action was not taken earlier (even two doubling times or six days earlier) is what is important. Had we done so, we would be in the same position as Germany. There is no point crying now – poor early decisions have disproportionate consequences, for cases, deaths and time to control the epidemic due to the explosive nature of transmission.
@Daren Austin
+1 Simon Anthony ignores difficult questions which show he is wrong
I posted one. Many hours after my Q, he replied to easy one from another directly above and ignored mine. I poked him for a reply – crickets
Happy to answer those. We are currently looking at about 75k excess deaths (all cause) for the first half of 2020 compared with the previous ten year average. A reasonable and conservative estimate is that lockdown has saved about two to three times that number of deaths (even assuming only 50% will eventually be infected with 0.1% fatality rate), and that better therapies (O2 and antivirals are forthcoming). In the near future.
I suspect the decision was motivated in some sense by panic at realising the extent cases and deaths were growing. Germany acted seriously when faced with the same information and are reaping the rewards now. We are paying for that. Once daily infections have fallen to manageable levels, Sweden shows (by some luck rather than judgement, I think) what steady-state looks like, with a progression to eventual endemic state. I don’t personally believe in herd immunity for coronaviruses, nor lasting protection from vaccines. I’d like to be proven wrong.
Even the Imperial group would not disagree with that point. Their premise was that every assumption about the progress of the epidemic predicted swamping of healthcare. The exact multiple was irrelevant. A better media communication might have been the severity of the situation shows that we will be inundated with cases within four weeks and urgent action is required, not an absolute numbers of lives will be lost.
Early epidemic predictions beyond doubling times are not possible for exponential growth without reasonably strong assumptions. Only after some deviation from exponential growth is better inference possible.
Comments are closed.