I have started looking into the way GCMs are used to produce regional climate predictions. The method seems to consist of weighting GCMs according to their regional concordance. I wonder if they are aware of the pitfalls of ‘chasing higher correlations’.
The most common approach has been to assess how well each of the available models simulates the present climate of the region (e.g. Dessai et al. 2005), on the assumption that the more accurately a model is able to reproduce key aspects of the regional climate, the more likely it is to provide reliable guidance for future changes in the region. The method of weighting models is presented shortly. From Climate Change in Australia 2007, Technical Report – Chapter 5: Regional climate change projections (temperature and precipitation)
Weighting of models
In preparing the CSIRO (2001) projections, simulations of present average (1961-1990) patterns of precipitation, surface air temperature (â€˜temperatureâ€™) and sea level pressure were examined. Models that performed poorly overall were not included in the projections. This approach is supported by recent research (Whetton et al. 2007). A similar approach to CSIRO (2001) is used in this report, except that all models are given a weighting (M scores in Table 4.1) based on current climate performance across the three variables, rather than omitting models that perform poorly. The weightings are described in section 4.2.3. These values are used to weight each model in the construction of the local response PDFs. Since the weights vary between 0.3 and 0.7, there is less discrimination of models from these than the CSIRO (2001) approach of using, in effect, weights of 0 (omit) or 1 (accept).
To see why this protocol is potentially a problem, note that regional concordance has nothing to do with the causes of global warming. The real danger is that the regional differences are produced purely by random variations between the models, so the selection models is by accident.
Even when the models show good correlation over the observed period, they quickly revert to random outside the period, a kind of global warming entropy. The average of the selected models is then the overall model bias. This is noted here:
In practice, the large number of models means that the final projections are not particularly sensitive to the inclusion of these non-uniform weights, as will be shown for annual precipitation. Note that the single set of weights is used for all variables and at each location, normalised by the number of models for which relevant data were available.
The problematic nature of correlation chasing is that it has no natural limit. If you select 50% of models that predict continental Australia best, then why not select the 20% that perform best in the Western Australian region and use them. Why not select the best model to use for predicting the weather at Perth? Why not a different model for each city?
The process could also be applied to variables. If 50% of models predict temperatures better, and the others predict rainfall, and still some others predict air pressure, then why not specialize them for each purpose? Why not select between models that predict mean rainfall and extreme rainfall. The problem is this leads to arbitrary results as noted here:
Naturally, weights chosen in a different manner, such as if performance over a smaller region was considered, may lead to somewhat different projected changes from those presented here.
I call it ‘accident chasing’. Ironically the higher correlation on the observed data, the lower the accuracy on unobserved (out-of-sample) data.
In reference to the Drought Exceptional Circumstances report, there is evidence the predictions of extremely low precipitation are particularly sensitive to the weighting scheme of the models.
The sensitivity of precipitation change probabilities to the tested uncertainties are region specific, but some conclusions can be drawn. At the 95th percentile, the uncertainty that tends to dominate is emissions scenarios, closely followed by GCM weighting scheme and the climate sensitivity. At the 5th percentile, GCM weighting scheme uncertainty tends to dominate for JJA, but for DJF all uncertainties have similar proportionate influence.
Dessai, S., X.F. Lu and M. Hulme, 2005: Limited sensitivity analysis of regional climate change probabilities for the 21st century. Journal of Geophysical Research-Atmospheres, 110, D19108.
I wonder if anyone has tested the assumption “that the more accurately a model is able to reproduce key aspects of the regional climate, the more likely it is to provide reliable guidance for future changes in the region.” This would consist of, say, comparing the difference in ranking of models over the periods 1900-1950 and 1957-2007. If the ranking is unchanged then well and good. If the ranking changes significantly, you have a problem. Regional concordance is not helping you.
Perhaps the statement in the IPCC Special Report on The Regional Impacts of Climate Change An Assessment of Vulnerability (1997) still holds true:
The IPCC has concluded that climate models at present provide useful predictions at the global and continental scale, but as yet allow little confidence at subcontinental scales (IPCC 1996, WG I, Section 6.6.3; Annex B of this report).