K.R.W. Brewer1 and A.N. Other1
28 January, 2009
1. K.R.W. Brewer is an Accredited Statistician of the Statistical Society of Australia Inc. (SSAI) and a long term Visiting Fellow at the School of Finance and Applied Statistics within the College of Business and Economics at the Australian National University.
2. A.N. Other is a pseudonym for another Accredited Statistician of the SSAI who prefers to remain anonymous. Full responsibility for the content is taken by K.R.W. Brewer.
The Drought Exceptional Circumstances Report (DECR) was authored by a team drawn from the CSIRO and Australiaâ€™s Bureau of Meteorology, and was publicly released in July 2008. Almost immediately it became a source of controversy. This evaluation, both of the Report itself and of the critique of it written by Dr David Stockwell, finds good mixed with less than good in both. The DECR itself is criticized for its poor delineation of Regions within Australia, for the choices made of statistics to be constructed, for the manners of their construction, and for not getting the best out of the relevant available data. Dr Stockwell is criticized for his inappropriate choices of methodology and of time periods for analysis, and also for misunderstanding some parts of what the DECRâ€™s authors had chosen to do. Nevertheless, both the Report itself and Dr Stockwellâ€™s critique of it are welcome stimuli to further investigate a serious issue within the climate change debate.
The Drought Exceptional Circumstances Report (DECR)  was commissioned by the Australian Bureau of Rural Sciences (an arm of the Australian Governmentâ€™s Department of Agriculture, Fisheries and Forests). It is a product of the Centre for Australian Weather and Climate Research (CAWCR), which is a partnership between the Commonwealth Scientific and Industrial Research Organisationâ€™s (CSIROâ€™s) Division of Marine and Atmospheric Research and the Australian Governmentâ€™s Bureau of Meteorology (BoM). It is also a product of the CSIROâ€™s Climate Adaptation Flagship. The Report was dated July 2008, and is officially titled, â€œAn assessment of the impact of climate change on the nature and frequency of exceptional climatic events.â€ Both that Report and the â€œSupplementary Informationâ€ for it, which was issued separately on 8 July 2008, have this title on their front covers.
Following the DECRâ€™s public release, a review of its analyses was undertaken by Dr David Stockwell, owner of the Niche Modeling blog, and posted there on 5 August 2008 . The results of this review were summarised in Ian Castlesâ€™ article â€œScientists, Politicians and Public Policyâ€ released on 7 August 2008 .
Dr Stockwellâ€™s criticisms of the DECR, were summarised by Mr Castles as follows:
o All 13 climate models failed internal validation tests for regional droughted area in Australia over the past century;
o Simulations showed increases in droughted area over the last century in all regions, while the observed trends in drought decreased in five of the seven regions identified in the CSIRO/BoM report;
o In almost all cases, the correlation coefficient between simulated and observed values was very low, and not significant;
o In all cases the â€œcoefficient of efficiencyâ€ was negative, indicating that the climate models simulated drought area worse than simply using the mean;
o In almost all cases the difference between the means of the return periods was significant, indicating that the frequency of droughts in the models has no relationship to the actual frequency of droughts;
o As the model simulations have no resemblance to observed droughts in the last century, the models have failed internal validation and no further testing is warranted;
o Contrary to statements in the report, there is no credible basis for claims of increasing frequency of Exceptional Circumstances declarations; and
o As there is no logical connection between the extreme values of models and simulations using different global warming scenarios, the reportâ€™s claim to have performed analysis using high global warming scenarios is illogical and invalid: this is true, irrespective of whether or not the underlying climate projections are themselves invalid.
Concluding his article, Mr Castles appealed to Accredited Statisticians to â€œâ€¦. carry out their own analysis of the data used in the CSIRO/Bureau of Meteorology study and to report the results. In doing so, they would be performing a valuable public serviceâ€¦.â€
As Accredited Statisticians interested (but impartial) in this debate, we are here responding to Ian Castlesâ€™ call for re-analyses. However, since this work has been conducted on our own time, and with limited resources, it is necessarily limited in scope. It is nevertheless our sincere hope that the comments we provide here will work as a catalyst for others, who have more resources than are at our disposal, to carry out further work to evaluate the CSIR/BoM report and related literature.
The principal reason for commissioning the DECR appears to have been the concern that recent and anticipated changes in climate within Australia have already led and would further lead to substantial increases in the numbers and extents of drought declarations in agricultural and pastoral areas throughout Australia, resulting in increased entitlements to assistance from the Australian Government in the â€œexceptional circumstancesâ€ (EC) defined under the current National Drought Policy (NDP). In the Terms of Reference in Appendix 1 to the DECR we find this wording:
To be classified as an EC event, the event must be rare, that is it must not have occurred more than once on average in every 20-25 years. Australia is experiencing a drought that has been unprecedented in its geographic extent, length and severity. Some areas have been drought declared for 13 of the last 16 years, leading to some recipients receiving EC assistance since 2002.
Climate change will bring significant challenges for Australian agriculture. Climate change is expected to increase the frequency, severity and length of drought periods in future.
Australian [presumably national, State and Territory] primary industry ministers have agreed that current approaches to drought and EC are no longer the most appropriate in the context of a changing climate. They agreed that drought policy must be improved to create an environment of self reliance and preparedness and encourage the adoption of appropriate climate change management practices.
The evident intent of these Terms of Reference was for the Reportâ€™s authors to describe what differences climate change was likely to make in the incidence and severity of drought events, in order to frame a new and more relevant approach to the support of farmers facing drought conditions. For instance, if obviously severe droughts were now or soon to be expected every ten years, the definition of an EC event as necessarily not occurring â€œmore than once on average every 20-25 yearsâ€ would need to be reconsidered. The DECRâ€™s Summary on p1 makes this point explicitly:
The current EC trigger, based on historical records, has already resulted in many areas of Australia being drought declared in more than five per cent of years, and the frequency and severity are likely to increase. The principal implication of this study is that the existing trigger is not appropriate under a changing climate. (Reviewersâ€™ emphasis.)
The Reportâ€™s authors responded to these Terms of Reference by providing projected changes in the incidences of exceptionally hot events, exceptionally low rainfall events and exceptionally low soil moisture events. In describing these changes they made use of a specific division into Regions. One of these was the Murray Darling Basin (or MDB, bounded by its watershed) and the other were six geo-political Regions, covering the whole of Australia, including the MDB, which were divided almost exclusively by State boundaries and the 26Â°S parallel of latitude. Apart from the MDB, the remaining six Regions were
New South Wales (NSW)
North-West Australia (NW) including the NT and WA north of 26Â°S
South-West Australia (SW) including SA and most of WA south of 26Â°S
Southwest Western Australia (SW WA) – roughly southwest of a line between
Geraldton and Hood Point – and finally
Victoria and Tasmania (Vic&Tas)
The main conclusions of the Report included the following four important passages (all verbatim quotes):
1. (p1) The uncertainties associated with the historical data and the climate projections are noted, including the qualitative assessment that the temperature data have the lowest uncertainty, that there is higher uncertainty with the rainfall data, and that the soil moisture data â€“ being derived from a combination of rainfall data, low resolution observations of evaporation, and modeling â€“ are the least reliable.
2. (p13) In summary, the analysis clearly shows that the areal extent and frequency of exceptionally hot years have been increasing rapidly over recent decades and this trend is expected to continue in future.
3. (p16) In summary, if rainfall were the sole trigger for EC declarations, then the mean projections indicate that more declarations would be likely, over larger areas, in the SW, SWWA and Vic&Tas regions for 2010-2040, with little detectable change in the other regions. Under the high scenario in all regions, EC declarations would likely be triggered about twice as often (at least four times as often in SWWA) and over double the areas (quadruple the area in SWWA).
4. (p17) In summary, projected increases in the areal extent and frequency of exceptionally low soil moisture are more evident than those for rainfall, notwithstanding the different calculation methods. … [However as] stated earlier, the soil moisture datasets and analysis have higher levels of uncertainty than those for temperature and rainfall.
Turning to David Stockwellâ€™s 2008 critique of the DECR, , we see (p2) that he chose to examine only the rainfall variable, explicitly â€œbecause low rainfall is a critical [input] to the EC definition of droughtâ€, but presumably also because the soil moisture data was the least reliable. Since his critique is confined to the rainfall data, we have chosen to make our comments also, and particularly our comments on his contribution, concentrate principally on the DECRâ€™s use of rainfall data. However, where other data are particularly relevant we will not feel inhibited about referring to them also.
2 The Rainfall Information in Figures 4a and 4b on Page 9 of the DECR
So far as rainfall is concerned, a highly informative part of the DECR is its page 9. Figure 4a on that page consists of a graph showing, for Australia as a whole, its annual rainfalls over the period 1900-2007, and Figure 4b is a map of Australia showing â€œTrend in annual rainfall 1950-2007 (mm per decade)â€.
The most noteworthy features of the graph in Figure 4a are as follows:
1. Looking at the trend line, there was remarkably little variability in rainfall over the entire 108 years on record.
2. The graph starts with a six-year period of comparatively little rain. (We have not checked this in detail, but it appears to be the driest period of six consecutive years in the data set.)
3. It also ends with a six-year period of comparatively little rain. (Again, we have not checked this, but it may well be the second driest period of six consecutive years in the data set. However, a third such dry period in the early 1960s would seem to run it close.)
4. The two periods, one in the mid-1970s and the other leading up to the turn of the millennium, both of which were both relatively rainy.
The fact that the period on record both starts and ends with a serious drought will be seen to be particularly relevant when we consider Stockwellâ€™s critique in detail (in Sections 7 and 8 below).
[Note that there are indeed 108 years of rainfall recorded on Figure 4b, 1900-2007. The gaps between the vertical lines narrow towards the last two â€œdecadesâ€ of the chart, giving the appearance that there are 109 years recorded. A more accurately drawn version of this figure appears on p35 of the Supplementary Information.]
In addition it was not stated whether the years shown were calendar or financial years. Both definitions are used in different parts of the Supplementary Information provided with the DECR. Compare, for example the first four graphs shown for Queensland on page 12 of the Supplementary Information document, for which that distinction is made, with the third graph on page 14: â€œAnnual Rainfall (mm) â€“ Queensland regionâ€, where no such distinction is supplied.
The most noteworthy features of the map in Figure 4b, also on Page 9, are as follows:
1. Its use of absolute reductions in rainfall (specifically in mm per decade), as opposed to percentage reductions, over the period 1950-2007.
2. The severe reduction in rainfall over that period in Australia east of the Great Dividing Range (the Divide).
3. A similarly severe, though far more local, reduction in rainfall in Southwest Western Australia.
4. The substantial increases in rainfall over that same period in the northeast of Western Australia and in the northwest of the Northern Territory.
5. The heterogeneity of the trends of rainfall pattern over that same period within each of the six States and also within the Northern Territory.
This map must be commended as being very informative with regard to the geographic diversity of the rainfall trends. It is, however, unfortunate that the trends depicted in this map were shown in absolute rather than percentage terms, as the severity of a reduction in annual rainfall tends to be experienced more in percentage than in absolute terms. For instance, a 50 mm reduction per decade in a region with 2500 mm in annual rainfall, such as might be found near Tully and Innisfail, would have much less impact than a 50 mm per decade reduction in a region with only 1000 mm annual rainfall, such as might be found close to Sydney or Melbourne. In consequence of this, Figure 4b (taken by itself) could give a misleading impression as to the uniformity or otherwise of the trends in Australiaâ€™s rainfall over the decades.
Nevertheless, Figures 4a and 4b taken together do suggest that local incidences of diminishing rainfall (chiefly in eastern Australia) could have been putting a severe strain on the users of that rainwater. Moreover, since the reductions in rainfall per decade (predominantly east of the Divide) have been accumulating since 1950, it seems not unreasonable to predict that they will continue, with consequences thereby becoming more severe in the foreseeable future.
In summary, our comments on Figures 4a and 4b are as follows:
1. Figure 4b, despite its shortcomings, shows a great deal of geographical variation in the rainfall trend over 1950 â€“ 2007.
2. Figure 4a completely ignores this geographical variation, and therefore does not provide any indication at all as to the severity of the rainfall reductions in certain regions.
3. Strong geographical variations in rainfall trend within most States and also within the Northern Territory make the regions defined in Figure 1 inappropriate units for rainfall analyses. To be of greater usefulness, regions should be defined so as to be reasonably homogeneous both in their current climatic conditions and in the trends of those conditions.
4. The absolute increases/decreases in rainfall per decade shown in Figure 4b are less useful than the corresponding percentage reductions would have been, had they been shown instead.
In addition, we would like to see more details as to how the trends in annual total rainfall depicted in Figure 4b were actually calculated.
In an over-all summary of the relevance of the rainfall data, we may note that while the analyses in Section 4.3 (pp16-17), based on the entire set of records over the period 1900-2007, do not suggest any great risk of the drought becoming more severe (other than in SWWA) the further information provided in Figure 4a, and more particularly in Figure 4b, does suggest that the rainfall trends over the period 1950-2007 might be both more relevant and more disturbing.
Finally, we note that these rainfall data would be more useful still if they were supplied for Regions that were more meaningfully defined in terms of geographical and climatological homogeneity.
3 Rainfall trends within the Murray-Darling Basin (MDB)
The MDBâ€™s boundaries follow the relevant watersheds, including most importantly the Divide along its eastern and southern boundaries, and it extends over parts of all States except WA and Tasmania. If there is any shortcoming whatever in its definition, it would have to be that the northern part of the Snowy River catchment was not included with it, because since the 1950s the water in that part of the catchment has predominantly been diverted into the rivers and storage reservoirs in the MDB. Better still, the Northern Snowy River catchment might have been treated as a separate Region on its own. (This catchment lies to the south-east of the MDB proper. The Snowy River itself runs mainly southwards through the â€œBalance of NSWâ€ and the â€œBalance of Victoriaâ€, outside of the MDB.)
Thus the problems facing the MDB and its rivers have at least three aspects:
1. Rainfall within the MDB proper has substantially reduced over 1950-2007.
2. It appears from Figure 4b that there has been an even greater reduction since 1950 in the rainfall within the Northern Snowy River Catchment, which for most of that period the Snowy Mountains Scheme has predominantly been diverting to the MDB.
3. The severely reduced amount of water now available to the MDB is (at least arguably) over-allocated to irrigation.
The DECR concentrates only on the first of these three aspects.
4 The Delineation of the DECRâ€™s other Regions (See Figure 1 on page 5)
As mentioned in Section 2 above, all six States and the Northern Territory have experienced quite heterogeneous rainfall trends within their boundaries over the period 1950-2007. It is therefore unfortunate that the DECR should have used State boundaries so freely in its delineation of the six Regions that between them cover the whole of Australia.
The principal motivation for the use of State boundaries to define the above regions would presumably have been that the relevant State Governments needed statistics directly relevant to them. However in practice the only States that could actually be provided with such figures would be New South Wales and Queensland. The other four States and the Northern Territory could not. It is therefore questionable as to whether State boundaries needed to be taken into account at all, especially since there is so much variability within most of those administrative areas as to mask the serious reductions of rainfall that have occurred within important portions of them over the decades since 1950.
We therefore suggest tentatively that the balance of Australia (outside the MDB) be divided into more natural Regions, each having a reasonably uniform rainfall distribution within it.
The actual division into more natural Regions would best be undertaken after a â€œrainfall trendsâ€ map had been drawn up resembling that already shown in Figure 4b, but showing percentage rather than absolute changes over the period 1950-2007. The temperature change map in Figure 2b should be taken into account as well.
This division is one that should be undertaken by people who are professionally familiar with the geography and the climates of Australia. Consequently the list shown below is not intended as a substantive suggestion, but merely as an indication of what a properly drawn up division of Australia into regions might look like.
The MDB within its presently defined boundary
The northern part of the Snowy River catchment, which feeds into the MDB
The balances of Vic, of NSW, and of southern and central Qld, south and east of the
Divide (either separately or combined)
The Cape York Peninsula
The balance of Qld west of the Divide
Central Northern Australia (Parts of the NT and of WA)
Northwestern Australia (Other parts of the NT and of WA)
Far western WA (SW WA extended northwards to Dampier or Port Hedland)
The balances of WA and of SA (either separately or combined)
Tasmania (possibly divided roughly into east and west).
5 The DECRâ€™s analysis of drought and Dr Stockwellâ€™s critique of it
The DECRâ€™s analyses of the impact of drought are described only in outline on p17 of the DECR. More information is given in the Supplementary Information, but some important details are still missing. However the â€œpercentage areaâ€ methodology evidently started by identifying (for each year and for each of the seven regions) those small individual areas (25km grid cells) that were in the highest five percentile groups for temperature over the period 1900-2007 (Table 4), in the lowest five percentile groups for rainfall over that same period (Table 7), or else in the lowest five percentile groups for soil moisture over the period 1957-2006, (Table 9).
For the identification process, the grid cells were aggregated in decreasing order of temperature or increasing order of rainfall or soil moisture until the cumulated percentage of total area in each region approximated five per cent of the regionâ€™s total area. These approximations resulted in the figures in the first columns of Tables 4 being somewhat less than 5% (for temperature), and those in Tables 7 and 9 being somewhat in excess of 5% (for rainfall, and also more severely for soil moisture).
The figures in the later columns of Tables 4, 7 and 9 indicate the percentages of total area projected to experience similarly high temperatures or similarly small amounts of rain or soil moisture in a later time period (2010-2040 for temperature and rainfall, 50 years centred on 2030 for soil moisture). The rates at which those areas are shown as growing or shrinking therefore reflect the extent to which the regionâ€™s drought conditions have been projected as becoming more or less severe.
The â€œreturn periodsâ€ methodology started somewhat similarly by counting (again for 25km grid cells within each region) the numbers of years between successive starts of droughts, a drought year being one in the highest five percentile groups for temperature (Table 5), the lowest five percentile groups for rainfall (Table 8) or the lowest five percentile groups for soil moisture (Table 10). â€œIf the length of a dataset were a multiple of 20 years and there were no repeated values, exactly 5% of the annual values would fall below the 5th percentile (or above the 95th percentile). However the lengths of our datasets used are not multiples of 20 years, hence there are small differences in the base rates and return periods for each datasetâ€ (DECR, p4 Box 2). In fact, the approximations used have resulted in the â€œreturn periodsâ€ for temperature exceeding 20 years and those for rainfall and for soil moisture falling somewhat short of 20 years.
All three measurements (temperature, rainfall, and soil moisture) were initially regarded as relevant to the definition of drought. However, as has already been mentioned in the concluding paragraph of Section 1, Dr Stockwell regarded rainfall in particular as â€œcritical to the EC definition of droughtâ€, and therefore limited his attention only to the rainfall measurements. Since Dr Stockwellâ€™s critique is the most detailed of those which have been made up till now, the current writersâ€™ comments will continue to concentrate on rainfall, or more accurately on the proportions of each Regionâ€™s area that would qualify to be drought declared, using the single criterion that they were currently in the lowest five percentile groups for their rainfall expectations, based on their own past history.
(This distinction between â€œrainfallâ€ and â€œdrought declaration based solely on historically low rainfallâ€ might potentially be confusing for anyone reading Dr Stockwellâ€™s critique without having previously readily read the DECR itself. The actual situation is as follows. In pages 13-16 of the original DECR document it is clearly indicated that the effects of temperature, rainfall and soil moisture on area under drought were each considered separately. In consequence there is no contradiction between the statement that â€œThis [i.e. Dr Stockwellâ€™s own] report only examines the rainfall variableâ€ and his uniform presentation of the results in terms of drought affected areas. These are, throughout, the areas affected with drought as indicated solely by changes in the rainfall variable.)
There is a further important consequence of this distinction. Since the areas under drought in the data files made available to Dr Stockwell (and thus also to the public) were defined as being those experiencing rainfalls in their driest 5% of years, those individual yearsâ€™ percentages of area under drought were often (and in fact more often than not) zero values. This meant that in the process of producing those annual series, a great deal of information was being ignored, with a consequent loss of precision in the results of the analyses. Since, in addition, the subsequent analyses were simple regressions of those â€œarea under droughtâ€ percentages against time, it also meant that the residuals from those regressions were all a long way away from being normally distributed, making the usual interpretations of those regression results appreciably less meaningful.
6. The Rainfall Records by Region
Figures 4a and 4b on page 9 of the Report have already been considered in Section 2 above. A reduced size version of Figure 4a is also displayed on p35 of the Supplementary Information, together with corresponding rainfall charts for the Reportâ€™s seven Regions on pp14, 17, 20, 23, 26, 29 and 32.
The Murray Darling Basin chart on p29 starts with three low rainfall years (1900-1902) recovers in 1903 but shows a steady fall from about 470mm to 400mm around 1940. There is a sharper rise to about 550mm by 1950, a fall to 500 by about 1958, a relatively steady period at that level until 2000, and finally a seven year drought averaging around the 400mm level.
The NSW chart on p17 also starts with the same three low rainfall years (1900-1902), recovers to about 500mm in 1903, remaining at that level until 1930 when it starts a decline into a fairly severe drought around the 450mm level, which drought lasts from roughly 1940 to 1950. It then hovers between 550mm and 600mm (with dips in the 1960s and 1980s) until the drought starts in 2001 or 2002, again averaging around 450mm.
The Queensland chart on p14, like that of Australia as a whole, starts and ends with six-year periods of low rainfall. For most of the relevant period, it has averaged around 600mm, but there were relatively wet periods in the 1950s, the 1980s and the late 1990s.
The Northwest Australia chart on p26 shows a very different pattern from that of most regions in Australia. Rainfall was relatively constant around 400mm from 1900 to 1973. The 11 years from 1974 to 1984 averaged around 550mm. Then the period 1985 through 1996 averaged only about 450mm, but from 1997 onwards the average has been the highest yet at about 600mm.
The Southwest Australia chart on p23 shows no evidence of an early drought and remains relatively flat around 250mm until the mid 1970s, which had three consecutive years over 350mm. There was another period of relatively wet conditions in the 1990s, around the 300mm level, but during the 2000s the region was back around the 250mm level.
The Southwest WA chart on p32 shows its rainfall as remaining fairly constant around the 700mm level until 1960, then a steady fall to the 600mm level by the mid 1900s. It again remains fairly constant at this reduced level until 2001, and 2002 is the first of a six year period of even lower rainfall, in the region of 550mm.
The Vic&Tas chart on p20 shows a long period at the 700mm level until the mid 1940s, when it starts to rise rather rapidly to 800mm by 1951. It falls to 750 by the early 1960s, rises to 800mm again by the early 1970s, falls to 750 again by 1980, recovers to 800mm around 1990, declines steadily to 700mm by 2000, and the trend is still downwards at about the same rate during the 2000s.
7. Dr Stockwellâ€™s Regression Analysis and his Choice of Period
As indicated by the findings in Section 6, rainfall in most of the Regions, the two exceptions being Vic&Tas and the relatively small SWWA, was as low or lower in the early 1900s than it has so far been in the current drought; whereas in all regions except the NW it was noticeably higher in the 1950s than in 2001-2007. In consequence, Dr Stockwellâ€™s choiceâ€”to use OLS to regress the changes in area under drought (as defined by rainfall) on time over the full 108 yearsâ€”left only very small changes to compare, with the not surprising result that the model trends (labelled mean-trend-e in his Table 1) were on the whole in the opposite direction to the actual trends. Had the comparisons been made over the period 1950-2007, it seems (from the Reportâ€™s Figure 4b) that a very different picture could well have emerged for that reason alone.
Even then, however, the DECR definitions of â€œregionsâ€, as areas within or closely related to administrative boundaries, might well have masked the much larger changes that might otherwise have been detected. For a proper evaluation, both the period would need to be changed to 1950-2007 and the Regions would need to be defined in a more appropriate fashion.
8. The Regression Analyses and the 13 Climate Models.
Dr Stockwell further chose to restrict his analyses to an internal validation of the DECRâ€™s projections, which had been made using a set of 13 Global Climate Models or General Circulation Models (GCMs). (A further advantage of dividing the series into two, as suggested in Section 7, would be that the earlier period, 1900-1950, could be used for internal validation, leaving the period 1950-2007 free for external validation.)
The OLS regressions that Dr Stockwell used for validation purposes were as follows. The regressor variable in each case was time in years, from 1900 to 2007. There were 14 regressand variables for each Region, one of observed values and the remaining 13 of projected values, one projection for each of the 13 GCMs. For the observed values he used each Regionâ€™s 108 actual annual percentages of area classified as being â€œunder droughtâ€ (on the basis of those areas currently experiencing less than their own 5th percentile for rainfall). For each of the projected values series, he replaced each actual observation value by the value that it would have taken if the actual rainfall had been that projected by the corresponding GCM.
From this he produced eight tables, Table 1 using all the 13 models as a group and with a row for each Region, and Tables 2-8 (one for each of the seven Regions separately) each table having 13 rows, one for each GCM.
As mentioned above, the principal analytical tool used by Dr Stockwell was Ordinary Least Squares (OLS) regression of droughted areas against time. However, as already mentioned in Section 5, we are not comfortable with the application of OLS to these data. The droughted areas were explicitly defined using a criterion of being in the lowest 5 percentile of rainfall. Not surprisingly, well over half of the individual observations had zero values.
But OLS is ideally intended for series which have normally distributed residuals, which is clearly not the case in this instance. Although OLS is robust, the obvious gross departures from normality throw strong suspicion over the validity of the usual tests of significance, and indeed the majority of the findings of the analyses.
Subject to those reservations above, we offer the following comments on Dr Stockwellâ€™s critique .
The principal findings from his Table 1 are those set out in (a) and (b) following :
Finding (a): â€œThe mean [climate model] trend for droughted area differ[ed] significantly from the observed trend in every region except Vic[&]Tas,â€
Comment: The claim of significance made here need to be considered in the context of the severe non-normality of the relevant residuals. The frequent occurrence of zero observations for droughted area in most of the regions occurs for the following reason. As explained in Section 5, the percentage of area under drought for any year in a given Region was based on the total over those small areas (25km grid cells) in that Region which were, in that year, experiencing rainfall in the lowest five percentile groups. In any given year that total could be anything from 0% to 100% of the Regionâ€™s total area. If the entire Region moved into and out of drought together, the proportion of zero observations would have averaged out as 95%. Because they did not in fact move together in that way, the proportion of zero observations for any given Region was considerably less than 95%, but it was still typically a substantial proportion of the 108 years used in the regressions, and quite enough to render the assumption of normal residuals inappropriate. In consequence the claim of significance made here must be treated with caution.
Finding (b): â€œ[T]he climate models [were] significantly biased in the opposite direction to observed drought trends.â€
Comment: In 5 out of 7 Regions the droughted areaâ€™s â€œtrend-oâ€ decreased marginally over the period 1900-2007, whilst in all Regions the 13 modelsâ€™ â€œmean-trend-eâ€ pointed to small increases over that same period. The observed slow declines in most Regions over the full period of 108 years appear to be largely attributable to the â€œFederation Droughtâ€ of the 1900s being somewhat more severe than the current â€œMillenium Droughtâ€. Had the regressions been restricted to the period 1950-2007, it is almost certain that no reversal in sign would have eventuated between the observed trends and the climate modelsâ€™ trends.
The largest such discrepancy by far was that in Queensland, where (in any case) all the models had failed spectacularly. For all the remaining regions, however, since the trend measures were quite small, the reversals in sign were of little account. The largest discrepancies between â€œtrend-oâ€ and â€œmean-trend-eâ€, other than those for Queensland, were +0.05 (one twentieth of a percentage point per year) in the MDB and in NSW. [We take the â€œvery low observed trend (+1% per year)â€ in Vic&Tas to be a slip for â€œ… plus one hundredth part of a percentage point per year.â€]
There is a further problem in the interpretation of Dr Stockwellâ€™s Table 1. It would appear that his sd-trend-e values are intended to be the standard deviations within a DECR region, as estimated using those 13 GCM series. But on this basis, the seven t-test values in his Table 1 would need to be approximately as follows: MDB 1.0, NSW 1.2, NWAust 1.3, Qld 2.4, SW-WA 0.5, SWAust 1.0 and VicTas 0.3. If this interpretation is correct, the corresponding p values would all be much larger, and only the Queensland value of p would be conventionally significant. There may be some other interpretation, but if so it is not an obvious one. (Admittedly this interpretation has been made using the properties of the normal distribution, but even in the absence of normality similar comments would almost certainly apply.)
In relation to his Tables 2-8 (one for each region) Dr Stockwell made three further findings:
Finding (c): â€œ[T]he models on average explained less than 1% of the observed variation in rainfall.â€
Comment: This appears to be a reference to the consistently small values in the r2 columns of Tables 2-8. If the implication is that the GCM-based projections do not reflect year to year changes in the drought affected percentages of the seven Regions, we do not regard this as a serious failure. It is not what the GCMs were constructed to do. They were meant to indicate long term trends.
Finding (d): â€œ[T]he climate models simulate drought area worse than the mean.â€
Comment: This is clearly a comment based on the consistently negative values of the Nash-Sutcliffe coefficient in Tables 2-8. It implies that â€œIf averaged over time, each of the 13 GCMsâ€™ sets of projections lies further away from the corresponding set of observed values than the simple mean of all those observed values does.â€ This result is probably another consequence of the â€œreversal in signâ€, which in our comment on Finding (b) above we showed to be itself a consequence of choosing for analysis a period over which there had been almost no net movement in rainfall.
Finding (e): â€œIn almost all cases the difference[s] between [the observed valuesâ€™ trend for the return period and the mean of the model trends projected values for the same concept] was significant. This indicates that the frequency of droughts in the models had no relationship to the actual frequency of droughts.â€
Comment: The manner in which the relevant series of droughted areas for each Region were calculated has been described in Section 5. To use it, the authors of the DECR would have needed access to (both observed and projected) rainfall data within 25km grid cells, which Dr Stockwell does not appear to have been supplied with. In the absence of these small area data, it would only have been possible for him to treat each Region as a whole, and determine for each year of record whether or not it was in one of its own five lowest percentile groups. But that is not the same calculation that would have been possible for the DECR authors, and the â€œfallacy of compositionâ€ effect makes it more than possible that the two calculations of return period could be widely different for that reason alone. If this is indeed what happened, that would explain the great preponderance of significant values for Dr Stockwellâ€™s â€œreturnp-pâ€.
In summary: Dr Stockwellâ€™s main conclusion from his analyses is that â€œall 13 climate models failed â€¦â€ Our view is that this conclusion is premature on account of the way his analyses were carried out. The percentages of area under drought as defined by extremely low rainfall are predominantly zeros. Such a data set is a long way away from being normally distributed, so not particularly amenable to OLS regression analysis, and is further subject to the loss of information described in the last paragraph of Section 5.
9. A Possible Alternative to OLS Regression.
It is at least possible that forecasting using simple ARIMA modeling , , might prove to be just as accurate and far easier to justify than OLS regression. ARIMA modeling is a purely statistical procedure which can be applied without any knowledge of the physical science underlying the phenomenon or phenomena being observed. The first and most essential assumption behind it is that the nature of the series is such that its future can reasonably be anticipated using only its past behaviour. Consequently, if the science behind a competing model is relevant, that model is likely to perform better than the ARIMA model. However, if it is not relevant, the ARIMA model, by ignoring that irrelevant science, is likely to be the better performer. In other words, ARIMA modeling is robust against climate model breakdown.
10. A Summary of Recommendations for a Possible Revision of the DECR Report
Although we are not at present convinced that Dr Stockwellâ€™s specific criticisms of the DECR are valid, we have serious reservations concerning that Report on our own account, which are implicit in the following recommendations.
1. The regions used in the Report are not optimally chosen. In their redefinition, more attention should be paid to the homogeneity of the geographical and climatological features within each of them. Correspondingly less attention should be paid to administrative boundaries. In addition, percentage changes in rainfall trends should be used, rather than absolute changes, when considering the importance of those climatological features.
2. The relevance of the Snowy Mountains Scheme in diverting rainwater from the upper Snowy River catchment area into the Murray Darling Basin needs to be recognized.
3. The three analyses, by temperature, rainfall and soil moisture respectively, could be retained, but the behaviour of the original data should form the basis of those analyses, rather than the changes in the percentages of area affected by drought. This is because a great deal of information is lost as a result of the required transformations. It would also be helpful if the revised report could provide more detail as to how the climate models were used to provide the projections, particularly the projections of return periods.
4. While the three analyses mentioned above are all potentially useful, the most relevant of the three is that by rainfall. If the rainfall analysis is done first and found to be successful, the other two might also be worth attempting.
5. If a fourth analysis is also undertaken, namely one by extent of drought affected area, then all three factors could initially be taken into account simultaneously. If this analysis is found not to be meaningful, that might obviate the need for it to be repeated for each factor separately. However, since so much detail is lost in the transformation, the results of any such analysis should be treated with more circumspection than those obtained using the original (observed) data.
6. Dr Stockwell has argued that the GCMs should be subject to testing of their adequacy using historical, or external data. We agree that this should be undertaken as a matter of course by all modellers. It is not clear from the DECR whether or not any such validation analyses have been undertaken by CSIRO/BoM. If they have, we urge CSIRO/BoM make the results available so that readers can make their own judgments as to the accuracy of the forecasts. If they have not, we urge them to undertake some.
7. If any such re-evaluation is to be carried out, however, it should be done using two separate time periods, namely 1900-1950 (during which the rainfall trend was generally upwards) and 1950-2007 (where it was generally downwards.) This would allow the earlier period to provide internal validations and the later period external validations. However, if and when these analyses are repeated, the raw data used should be compiled not for the existing seven Regions, but for more homogeneous Regions, as suggested in item 1 above.
 Hennessy, K., Fawcett, R., Kirono, D., Mpelasoka, F., Jones, D., Batholsa, J., Whetton, P., Stafford Smith, M., Howden, M., Mitchell, C. and Plummer, N. An assessment of the impact of climate change on the nature and frequency of exceptional climatic events. Technical Report with a separate document entitled â€œSupplementary Informationâ€, CSIRO and the Australian Bureau of Meteorology, for the Australian Bureau of Rural Sciences, July, 2008.
 Stockwell, David R. B. â€œTests of Regional Climate Model Validity in the Drought Exceptional Circumstances Report.â€ Niche Modeling (http://landshape.org/enm).
 Box, G.E.P. and Jenkins, G.M., (1976). Time Series Analysis: Forecasting and Control, Holden-Day: San Francisco.
 Pankratz, Alan, (1983). Forecasting with Univariate Box-Jenkins Models, Wiley:New York.