As a follow-up om the previous post, I have examined the correlation statistics for the reconstruction of past climate from random series with red noise. I have tried to use the same approach as MBH98, where the model is tested over data for years held back from the main analysis and model development. Different intervals of years could be chosen, but in the case of MBH98, the model is trained on years 1901-1990 and tested on years 1856-1900. The distribution of R2 values are as follows:
Figure 1. The frequency distribution of R2 values for all series (trees) over the training interval in blue, and the test interval in red. The distribution of R2 before selection is shown by the solid line and after selection by the dashed line. Series are selected if the R2 value is greater than 0.1 and have a positive slope.
The figure above shows the shift in R2 values of the series to the right resulting from the selection for significant correlation. Most of the R2 values for the individual series however are below the cutoff value of 0.1 which can be regarded as not significant. This indicates the cross-validation statistics on the individual random sequences have no skill at predicting temperatures on the test data. However, the table below shows the R2 for the raw and smoothed versions, where CRU is the raw temperature data and CRUgs is the data smoothed with a 50 year Gaussian filter.
|Training period CRU~recon||0.56|
|Test period CRU~recon||0.002|
|Training period CRUgs~recon||0.91|
|Test period CRUgs~recon||0.44|
While the correlation of the reconstruction with raw temperature data is very low, the correlation with the smoothed temperature data is very high. In fact, the R2 value over the supposedly independent test period from the Gaussian smoothed data is a highly significant 0.44. Thus it could easily be claimed that the reconstruction has skill at predicting the temperatures from the held-back period. The correlation can be clearly seen on the reconstruction below:
Figure 2. The reconstruction of past temperatures using selected random sequences, with at training period of 1901-1990 and a test period of 1856-1900. The test period corresponds to the lower ‘hook’ in the CRU temperatures (heavy line) below, and the corresponding hook in the reconstruction is clearly visible.
Relevance to other studies
While this validation is based on the division of temperature data into years as used in the MBH98 study, it could easily be applicable to other choices of periods and validation protocols (http://www.climateaudit.org/?p=557). Moreover, these results are relevant to claims of robustness of MBH98 to variations in methodology (e.g. AW06). While AW06’s claims of robustness for MBH98 are not disputed here, robustness may not be a very severe test, or any kind of test at all, given it is possible to produce similar results irrespective of whether the individual tree-rings data are related to temperature. It must be remembered that what is important is not tested in the AW06 study: correct reconstruction of past temperatures. Obviously such tests are not directly possible as no instrument record for the entire reconstructed period exists. However, the reliability of different types of proxies might be obtained by comparing the results of different types: bore-holes, sediments, corals, tree-rings etc.
These results show that high cross-validation correlations can be obtained between a reconstruction and the smoothed versions of temperature even when the individual series and the reconstruction have no apparent skill at predicting held back unsmoothed temperature data. They also show a great deal of care needs to be taken to ensure validation tests truly reflect the skill of the reconstruction, particularly when dealings with smoothed data. The use of simulated random data in place of real tree-ring data illustrates that heavy reliance on individual selection or trees (a.k.a. cherry-picking) and individual calibration in this class of climate proxy opens the door to criticisms of circularity.
MBH98 – M. E. Mann, R. S. Bradley, and M. K. Hughes. Global scale temperature patterns and climate forcing over the last six centuries. Nature (London), 392(23), 1998.
AW06 – E.R. Wahl, and C.M. Ammann. Robustness of the Mann, Bradley, Hughes reconstruction of northern Hemisphere Surface Temperatures: Examination of Criticisms Based on the Nature and Processing of Proxy Climate Evidence. Climate Change, in press.