The ordinary eyeball test is a reliable and widely used form of data analysis, particularly in climate science. The basic approach is to plot data of a graph, use a highly complex and incompletely documented method to make it almost impossible to replicate, then visually present the desired results with a thick red line.
Experience has shown that the OET produces reliable results in almost every situation. Moreover, these results are very convincing. So convincing was the OET known as the Hockey Stick that it was used in press releases of Third Assessment Report by the United Nations’ Intergovernmental Panel on Climate Change (IPCC) in 2001, featured in the award winning movie “An Inconvenient Truth”, and is still widely used throughout government and science. It has rendered yeoman service in garnering acceptance for the notion that temperatures are the highest they have been for a million years.
Figure: Example of the Ordinary Eyeball Test (OET) from Rahmstorf et al. 2007.
One of the most recent examples of the OET is in the publication by seven IPCC lead authors (the Rahmstorf7) of Rahmstorf et al. 2007 in Science. Here an enhancement of the OET called the ‘slide and eyeball method‘ was used as the main scientific support for the notion of a present ‘runaway warming’ using a classic OET method to demonstrate that “the climate system is responding faster than our current generation of models suggest”. So convincing was this OET that citing this piece of evidence alone, the Australian Garnaut commission determined that “Developments in mainstream scientific opinion … suggest that the world is moving towards high risks of dangerous climate change more rapidly than has generally been understoodâ€. Recent economic modeling shows that implementing a 90 per cent reduction in carbon emissions by 2050 would alone require closing 3 out of 4 coal fired power stations, and building new clean power stations at a cost of $50 billion dollars.
Unfortunately the OET is vulnerable to attacks from the numerate, particularly the claim that uncertainty has not been properly accounted for. Some of the curve has been taken off the Hockey Stick with subsequent studies, but this has not prevented it from remaining in widespread use. The OET is usually acceptable to reviewers in Nature and Science. Some concerns have been raised in a submission to the Garnaut commission pointing out that a more arcane part of the methodology stems from the same person who developed Hockey Stick (Michael Mann) and that it advocates an even more extreme position than the already quite extreme global warming consensus of IPCC. In responding to questions about the uncertainty in the OET of the Rahmstorf7 at RealClimate.org, Rahmstorf seemed confused about the operation of the methodology itself, and suggested concerns be taken to peer review. It never pays to look too closely at an OET.
There are a few of the opinion that even papers that have passed Science and Nature’s peer-review processes, and scientific reports such as the IPCC approved by more than 100 governments, could benefit from review of the methodology by at least one expert statistician. But then the statistical deficiencies would be brought to wide public notice and much of the effectiveness of the OET would be lost.
Thanks to Peter Gallagher for the OET.