Correctly taken to task by two readers and some breaking news

I should have amended the previous post to say I mistrust unverified models.  Here are two comments

#1 Andyextance

  • “Leaving aside the questions of the reliability of models in different subjects, and whether all of your six reasons truly relate to models, I have one core question: Without models, how can we have any idea about what the future might hold? Models may not always be right – but as long as they have some level of predictive skill they can often at least be a guide.”

    Absolutely correct — it’s all about prediction, not plausibility.

#2 Former Bell Labs denizen

“And yet you board a commercial airliner without hesitation, freely trusting your life to the models of aerodynamics, materials science, control system theory, electronics, etc. that were used in designing the aircraft. Similar comments apply to entering a modern skyscraper, or even pushing the brake pedal on your automobile.
Perhaps what you are really saying is that you don’t trust models until their correctness is demonstrated by experience; after that, you trust them. Hey, nothing to disagree with there.”
Correct again
Breaking news
This just in — too late for yesterday’s post — the climate models have overestimated the amount of warming to be expected this century — the source  is an article  in
Nature Geoscience (2017) doi:10.1038/ngeo2973 — behind a paywall — but here’s the abstract
In the early twenty-first century, satellite-derived tropospheric warming trends were generally smaller than trends estimated from a large multi-model ensemble. Because observations and coupled model simulations do not have the same phasing of natural internal variability, such decadal differences in simulated and observed warming rates invariably occur. Here we analyse global-mean tropospheric temperatures from satellites and climate model simulations to examine whether warming rate differences over the satellite era can be explained by internal climate variability alone. We find that in the last two decades of the twentieth century, differences between modelled and observed tropospheric temperature trends are broadly consistent with internal variability. Over most of the early twenty-first century, however, model tropospheric warming is substantially larger than observed; warming rate differences are generally outside the range of trends arising from internal variability. The probability that multi-decadal internal variability fully explains the asymmetry between the late twentieth and early twenty-first century results is low (between zero and about 9%). It is also unlikely that this asymmetry is due to the combined effects of internal variability and a model error in climate sensitivity. We conclude that model overestimation of tropospheric warming in the early twenty-first century is partly due to systematic deficiencies in some of the post-2000 external forcings used in the model simulations.
Unfortunately the abstract doesn’t quantify generally smaller.
Models whose predictions are falsified by data are not to be trusted.
Yet another reason Trump was correct to get the US out of the Paris accords— in addition to the reasons he used — no method of verification, no penalties for failure to reduce CO2 etc. etc.  The US would tie itself in economic knots trying to live up to it, while other countries would emit pious goals for reduction and do very little. 
In addition, \ I find it rather intriguing that the article was not published in Nature Climate Change   –, — which would seem to be the appropriate place.  Perhaps it’s just too painful for them.
Post a comment or leave a trackback: Trackback URL.


  • Slava Bernat  On June 22, 2017 at 3:54 pm

    I’m not a climatologist but this sentence from the main text of the paper tells a lot about the quality of analysis they performed: “For the statistic gauging the asymmetry in the numbers of positive and negative difference series trends, p_gamma1 ~ 0.005. On average, therefore, there is only a 1 in 200 chance that the actual preponderance of significant positive trends in ΔTf−o(k, t) could be due to internal variability alone (Fig. 4a).”

    Similar statement is in the abstract: “The probability that multi-decadal internal variability fully explains the asymmetry between the late twentieth and early twenty-first century results is low (between zero and about 9%).” Where 9% is actually taken from p = 0.09.

    This is such a gross misinterpretation and misuse of the p-value that I wonder how would anyone take this seriously.
    Just eyeballing the figure 1 is enough to get the feeling of ‘overestimation’ magnitude, which is, well, not worth to write a paper about. There was a ‘substantial’ underestimation just before 2000, and then overestimation in early 2000s. But hell yeah, there was clearly something wrong with the model in the beginning of XXI century.

  • andyextance  On June 24, 2017 at 5:18 am

    I haven’t read the statistics part of the paper in close detail, and statistics are not my strong point, so I’m not sure if what you say is true Slava, but I’d be surprised if these authors were grossly misusing the p-value. The likes of Meehl, Santer and Hawkins are highly respected in the climate community (often published in Nature Climate Change). Can you explain why you can’t use the p-value to represent probability in this way?

    Yet, for me the Nature Geoscience paper doesn’t falsify climate models. Look carefully at the conclusion “systematic deficiencies in some of the *post-2000 external forcings* used in the model simulations”. What’s a forcing? It’s a factor that influences energy flow into or out of the climate. My understanding is that this is the data that is used to feed into a model. This is a tricky thing to pin down – a few years back there was an bit of debate over ‘missing heat’, the imbalance between the measured energy coming into the planet and the energy budget as measured. To quote from the paper: ‘These errors arise in part because the simulations were performed before more reliable estimates of early twenty-first century forcing became available20, 27. The net effect of the forcing errors is that the simulations underestimate some of the cooling influences contributing to the observed ‘slowdown’.’

    So the way I read the paper’s conclusion, is that it’s a case of ‘If your input data’s a bit off, your output’s going to be a bit off.’ It would be tempting to say garbage in, garbage out, but the models are not completely wrong – they were just a bit high for a few years. Carbon Brief has an interesting detailed analysis on this that’s worth a look: I note that the Daily Caller also picked up this paper as evidence that models are all borked, which in turn was picked up by Breitbart. CarbonBrief has a direct quote from one of the authors, the others do not. Working a science journalist, that’s an important indicator for me as to who’s going to the effort to get the technical details right.

    I try to be aware of my bias, and know that I routinely come down on the side of climate scientists, but I need to point out the logic surrounding much of the distrust of climate models. It tends to be: They don’t work perfectly, so we shouldn’t use them. The first part of that is certainly true, but the second half isn’t. Understanding models’ flaws is part of the verification process you seek. I don’t know, but would expect the models cited by Former Bell Labs denizen to have some problems, which the users are aware of and take into account. So scientists use models carefully – that was definitely the case when we used protein docking models when I worked in drug discovery. We also – as in this Nature Geoscience paper – try to understand why they’re a bit off, so that we can improve them.

    • Slava Bernat  On June 26, 2017 at 1:45 am

      OK, here’s my lay understanding.
      The most obvious problem with p values is that they don’t have much sense without effect size.
      The other problem with their analysis, and they do admit it, is that they treat some variables as independent, although they are likely depend on each other. This can easily lead to cubic (x^3) inflation of observed p-values.
      One more flaw that I suspect is also trivial and can be summarized by quoting one professor I used to work with: “where are the error bars?” Comparisons are made between mean values of observed and predicted data, but both have error and I didn’t notice how they treat that.
      And the problem that I’m the least sure about is in their methodology. I’m quite suspicious about using resampled past observations as a ‘null’ distribution to make comparison with. Any result (i.e. p value) coming from it gives one two explanations: either the null hypothesis can be rejected, or one deals with quite unusual sample and the null hypothesis is true. Under frequentist paradigm it’s impossible to find out which one is actually true. However, many researchers misinterpret the p-value as a probability of the second outcome. They could use Bayesian methods to find actual probability (with credible interval) that climate models are biased toward overestimation vs. observed data. To illustrate this point, consider throwing a die 5000 times. There will be periods when you get, say, mostly 4’s, 5’s and 6’es, so they’ll draw the average upwards from expected 3.5 value. Let’s stop observations at one of these lucky streaks. Now, if you take those periods with some previous observations, and draw 1000 random subsamples, and make nice histograms, you’ll see that they are much closer to uniform distribution and expected mean of 3.5 than those streaks of 4-5-6 observed in the end. Taking enough subsamples will get p-value low enough to conclude that those streaks are not random. But using Bayesian statistic, which will include past observation and some basic assumption about the die (i.e. it’s a ‘fair’ die), one can calculate the probability that the die is ‘loaded’, and it will likely be something like .33+/-0.1, which is much less impressive than a p-value of 0.01 or so. Obviously, the climate is more sophisticated than throwing a die, but I hope I could convey the idea.

      And of course, I agree that the paper doesn’t invalidate climate models (and doesn’t attempt to). It just shows that there might be something that we don’t take into account yet.

  • luysii  On June 26, 2017 at 11:19 am

    What I think is so significant about this paper is the admission, that the models have over predicted the amount of warming and thus might be incomplete by Mann and Santer (who are very big names in climate science). This doesn’t invalidate climate models and just shows that, like any model of reality that climate models need more work.

    Unfortunately, climate models have been treated as holy writ by political types pushing the Paris accords, who have been using them to push other agendas.

    Chemists have long used models — Einstein’s work on Brownian motion in 1905 was the first convincing proof that atoms actually existed. Before than atoms were just models about which there was considerable skepticism (Ernst Mach).

    My undergraduate advisor (Paul Schleyer) fought a war against H. C. Brown (and lost as he never got the Nobel) trying to extend the model of the carbon carbon bond to the nonClassical carbonium ion.

    • Slava Bernat  On June 26, 2017 at 12:51 pm

      I hate to sound like a climate activist but it kinda makes sense to take precautionary actions before all possible information is at hand. I don’t think one needs to prove harm from emissions to restrict them, but rather to prove their harmlessness if we are not to enforce the restrictions.

  • Data Geek  On March 5, 2019 at 4:44 am

    See link:

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: