Evidence Based Practise

Writing on Prof. Garnauts Heinz Arndt Lecture, Peter Gallagher pens a sensible comment on the lack of attempt to strike a balanced risk assessment. Unlike the 1996 inaugural speech of Adrian Smith as President of the Royal Statistical Society, who held out evidence-based practices as an exemplar for all public policy, Prof Garnaut dismissed the conflicting scientific evidence for global warming.

Evidence-Based Practice uses techniques from science, engineering, and statistics, such as meta-analysis of literature, risk-benefit analysis, and independent tests. EBP aims for clear understand of the relative quality of evidence used in decisions.

Generally, there are three distinct, but interdependent, areas of EBP. The first is the application of the most well-evidenced studies. This requires a basis for judging best research by some objective critieria. Thus second area is the systematic review of literature to evaluate the best studies. Finally, EBP is a “movement” where advocates work to popularize the method and usefulness, both by highlighting instances of good and bad evidence-based practise.

Evidence-based medicine has demoted statements of the “medical expert” to the least valid form of evidence. Thus, statements by the worlds leading organizations would rank poorly as a basis for public policy unless they are based on an evidence-based approach.

Systems to stratify evidence by quality have been developed, such as this one by the U.S. Preventive Services Task Force for ranking evidence about the effectiveness of treatments or screening:

Level I: Evidence obtained from at least one properly designed randomized controlled trial.
Level II-1: Evidence obtained from well-designed controlled trials without randomization.
Level II-2: Evidence obtained from well-designed cohort or case-control analytic studies, preferably from more than one center or research group.
Level II-3: Evidence obtained from multiple time series with or without the intervention. Dramatic results in uncontrolled trials might also be regarded as this type of evidence.
Level III: Opinions of respected authorities, based on clinical experience, descriptive studies, or reports of expert committees.

A more general approach from the Oxford Centre for Evidence-based Medicine might be more adaptable to earth sciences in general:

Level I: Well designed, randomised controlled trial, a decision rule validated in different conditions.
Level II: Well designed, comprehensive and independently tested studies; or extrapolations from level I studies.
Level III: Hypothesis tested theories or models with observations, or extrapolations from level II studies.
Level IV: Computer simulations, observational, based on physical principles, bench research or first principles.
Level V: Expert opinion without explicit, independent critical appraisal.

The difficulty with climate science and economics is the virtual impossibility of level I randomised controlled trials. This serves as a reminder the quality of evidence is generally never above level III. Human ingenuity may develop ways to improve the conduct of natural experiments, and the classification scheme serves to motivate better generation of evidence.

Based on the scheme above, Garnaut’s statement that the recommendations of the IPCC:

“Is not contested by the large majority of specialists, and by the leaders of the relevant learned academies in the countries” (page 6)

would put the evidence on level V, even though individual research examined by the report could occupy higher levels.

Consider other cases of sources of evidence, Global Climate Models (GCMs) and Paleoecological reconstructions.

GCMs would generally sit at level IV, one level higher than expert opinion, and so present low quality evidence. As an illustration that this evidence is indeed low quality, consider the Climate Impact Assessment (ACIA) report by an international team of 300 researchers for the Arctic Council, predicting the Arctic will lose 50% to 60% of its ice distribution. However, Arctic ice extent has since returned to long term averages, justifying its classification as level IV poor evidence of global warming.

Palaeoecology includes reconstructing past climates from tree-rings and other proxies of climate. The origin of the famous hockey stick graph claiming temperatures are the highest experienced in 600 (and then 1000) years, and this could only be attributed to human emission. This view on historic temperatures, particularly in the Medieval Warm Period, has been reversed by a number of more rigorous studies. At best, one could say that some of these studies sit at level III, and some at level IV.

When judged against these standards, one may well ask just where is the reliable evidence of global warming? What can be confidently attributed to human factors, and represents a serious threat? At the very least, the quality sources of evidence and their confidence would be clearly identified in an EBP.

Categories of recommendations

Recommendations for action on global warming, as Prof. Garnaut is doing, should be based on balance of risk versus benefit and the level of evidence on which this information is based. These levels, drawing on the U.S. Preventive Services Task Force are:

Level A: Good scientific evidence suggests that the benefits of the policy substantially outweighs the potential risks. Clinicians should discuss the service with eligible patients.
Level B: At least fair scientific evidence suggests that the benefits of the clinical service outweighs the potential risks. Clinicians should discuss the service with eligible patients.
Level C: At least fair scientific evidence suggests that there are benefits provided by the clinical service, but the balance between benefits and risks are too close for making general recommendations. Clinicians need not offer it unless there are individual considerations.
Level D: At least fair scientific evidence suggests that the risks of the clinical service outweighs potential benefits. Clinicians should not routinely offer the service to asymptomatic patients.
Level E: Scientific evidence is lacking, of poor quality, or conflicting, such that the risk versus benefit balance cannot be assessed. Clinicians should help patients understand the uncertainty surrounding the clinical service.

A large and growing body of scientists would suggest that despite all of the research, global warming is at Level E:

“the scientific evidence is lacking, of poor quality, or conflicting, such that the risk versus benefit balance cannot be assessed.”

The advice to Clinicians in this case is that they

“should help patients understand the uncertainty surrounding the clinical service”.

Ross Garnaut suggests that we should take up Pascal’s Wager out of fear of the possible consequences of global warming. A parallel can be drawn as in environmental science, as in medical science, where the treatment has attendant risks (costs). Garnaut wants us to take a bitter pill, costing at least $45 billion at this time. And while Garnaut acknowledges that

“The scientific assessments are highly uncertain, and their impacts on human activity and welfare even more so.” (page 4)

It does not follow that we are forced to take it:

“We have no alternative to making decisions on complex issues of valuation under great uncertainty.” (page 4)

While it is defensible to argue for taking the pill on the basis that public opinion wants and demands the treatment, evidence-based practise suggests that Garnaut should pay far more attention to understanding the uncertainties surrounding global warming and explaining these uncertainties to the public.

Adapted from wikipedia

Advertisements

21 thoughts on “Evidence Based Practise

  1. David, ‘evidence-based medicine is an interesting and pertinent framework for considering the obligations of policy-makers. Thank you for pointing to its relevance. The systems of the “Preventative Services Taskforce” seem especially apt.

    I suspect that RG would respond that the analogy is weakened by the time-scales involved in gathering data about the climate (hence the ‘wager’ element in his recommendations). But that does not really change the conclusion that the nature of the evidence and of the assurance (even his assurance, as you note) does not warrant drastic action.

    I think the only responsible course for RG would be to “hedge the bet” on warming by recommending measures more proportionate to Level V (or Level E in the clinical practice scale) as a start until we have a better understanding of the physics of climate (warming, for example) and need to rely less on probabilistic techniques based on time-series analysis to project future climate.

    But I’m not optimistic that this will be his recommendation.

  2. David, ‘evidence-based medicine is an interesting and pertinent framework for considering the obligations of policy-makers. Thank you for pointing to its relevance. The systems of the “Preventative Services Taskforce” seem especially apt.

    I suspect that RG would respond that the analogy is weakened by the time-scales involved in gathering data about the climate (hence the ‘wager’ element in his recommendations). But that does not really change the conclusion that the nature of the evidence and of the assurance (even his assurance, as you note) does not warrant drastic action.

    I think the only responsible course for RG would be to “hedge the bet” on warming by recommending measures more proportionate to Level V (or Level E in the clinical practice scale) as a start until we have a better understanding of the physics of climate (warming, for example) and need to rely less on probabilistic techniques based on time-series analysis to project future climate.

    But I’m not optimistic that this will be his recommendation.

  3. systematising types of evidence is very useful. Teh difficulty with climate seems to be that, like economics, you rarely get to do trials, let alone randomised trials. This suggest that the quality of the evidence will never rise above level III except in the case of “natural” experiments. We may of course now be conducting such a natural experiment, but we have to wait a long time before teh answer is in. Hence the problem.

  4. systematising types of evidence is very useful. Teh difficulty with climate seems to be that, like economics, you rarely get to do trials, let alone randomised trials. This suggest that the quality of the evidence will never rise above level III except in the case of “natural” experiments. We may of course now be conducting such a natural experiment, but we have to wait a long time before teh answer is in. Hence the problem.

  5. That is the nature of observational sciences, so never rising above level III is a worthwhile reminder just how much evidence is needed to provide certainty comparable with experimental sciences. I should put a paragraph in the post about that, as I hadn’t overlooked the issue.

    Another issue is that there are fewer issues in climate science, while clinical medicine is very ‘fine grained’. We are really looking at one supposed problem with a few symptoms, and few possible treatments.

  6. That is the nature of observational sciences, so never rising above level III is a worthwhile reminder just how much evidence is needed to provide certainty comparable with experimental sciences. I should put a paragraph in the post about that, as I hadn’t overlooked the issue.

    Another issue is that there are fewer issues in climate science, while clinical medicine is very ‘fine grained’. We are really looking at one supposed problem with a few symptoms, and few possible treatments.

  7. David, the “evidence-based medicine” slogan and many articles originated in southern Ontario, with one of the key authors, Gordon Guyatt, even coming from a squash-playing family. Something in the water, I guess. I’ve read some of the articles and there are some surprising parallels to issues in the multiproxy climate articles. Guyatt is mentioned in a couple of early CA comments http://www.climateaudit.org/?p=110#comment-577 ff. Cheers, Steve

  8. David, the “evidence-based medicine” slogan and many articles originated in southern Ontario, with one of the key authors, Gordon Guyatt, even coming from a squash-playing family. Something in the water, I guess. I’ve read some of the articles and there are some surprising parallels to issues in the multiproxy climate articles. Guyatt is mentioned in a couple of early CA comments http://www.climateaudit.org/?p=110#comment-577 ff. Cheers, Steve

  9. It never hurts to remember that it was real people who brought about such improvements, even when the size of the opposition, such as in climate science, is large. It is a positive direction for climate skeptics, and doable to implement something like this. People can understand these levels and can even classify the quality of papers even when they don’t grasp the statistical details. But I don’t think it would happen unless the perception of the IPCC changed from acquiescence to scandal (like oil-for-food).

    I think there are a few more posts in this topic. Cheers

  10. It never hurts to remember that it was real people who brought about such improvements, even when the size of the opposition, such as in climate science, is large. It is a positive direction for climate skeptics, and doable to implement something like this. People can understand these levels and can even classify the quality of papers even when they don’t grasp the statistical details. But I don’t think it would happen unless the perception of the IPCC changed from acquiescence to scandal (like oil-for-food).

    I think there are a few more posts in this topic. Cheers

  11. Pingback: dietetyka

  12. Pingback: link

  13. Pingback: strona www

  14. Pingback: tutaj

  15. Pingback: witryna

  16. Pingback: opieka informatyczna gliwice

  17. Pingback: zobacz

  18. Pingback: kliknij

  19. Pingback: filmowanie Lublin

  20. Pingback: magnificent

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s