Following the line of interest in the previous post on Evidence-based Practise, I came across a review of research into bias in clinical medical research by Lise Lotte Gluud summarizing the findings of methodological studies on the influence of bias in clinical trials. There are many learnings here that could be applied to climate and environmental sciences in general.
It is recognized that uncontrolled observations can provide reliable evidence if the effects are dramatic. However, great care must be exercised when the effects are moderate or small, as such effects as errors, bias, spurious correlations, limit the value of uncontrolled observations. Computer models are essential where direct experiments are impossible. The main problem of experiments with models lies in extrapolation to situations without experimental testing (i.e. predictions), as these may lead to the wrong conclusions.
Thus the main concern of evidence-based practise is detecting and controlling research bias. One definition of bias is the unknown or unacknowledged error created during the design, measurement, sampling, procedure, or choice of problem studied.
Bias is so pervasive because we want to confirm our beliefs, even though science should be organized around proving itself wrong. The key difference between evidence-based and other research is the explicit attempt to eliminate sources of bias.
Selection bias occurs when confounding factors are unevenly distributed between the experimental group and the control group. Selection bias is often called ‘cherry picking’, where only those data points with favorable outcomes are included in the experimental group. The medical field uses randomization to reduce such bias by creating control groups that are similar with regard to known confounding variables. For example, cores from trees selected for a dendrochronological study could include cores from a random sample of trees and subjected to parallel analysis. Climate analysis from a selected set of climate stations could include analysis of a random sample of stations.
Evaluation of models should be performed blind, where for example, the fit of climate models could be compared without the operators knowing which are the ‘real’ data, or what results were achieved by other models. Certain statistical methods such as logistic regression has been found to increase bias due to misclassification and measurement errors in confounding variables. So care should be taken in selecting methods that minimise inbuilt biases.
Medical science has regognized that the large randomized trial is one of our most reliable sources of evidence for assessment of intervention effects. As trials are not possible in the study of large systems such as climate, alternatives for capturing possible bias must be sought.
Adequate randomization requires that the results be uncertain (the ‘uncertainty principle’). If the result of a study is predetermined due to the study design or methodology, then bias compromises the results. Adequate randomization may be achieved by ‘monte carlo’ simulation, by the generation of random sequences and giving the random sequences and equal chance of producing the result.
The effect of competing interests is debated in medical research. It has been found that industry funding has been associated with higher quality than trials without external funding. On the other hand, financial interests may bias the interpretation of trial results.
The reason for the association between funding and pro-industry conclusions include violation of the uncertainty principle, publication bias, and biased interpretation of trial results. The uncertainty principle means that there is demonstrated uncertainty about the results of the study. Violation of the uncertainty principle may be related to ‘cherry picking’ or flawed methodologies.
In 1997, approximately 16 percent of 1,396 highly ranked scientific journals had policies on conflicts of interest (78). Less than 1 percent of the articles published in these journals contained disclosures of personal financial interests. The importance of disclosure of financial interests is increasingly being recognized, as demonstrated by the following examples of publication bias and neglect of harm.
Publication bias is the selective publication of the findings of trials with certain results, and may lead to exaggeration of effects. Medical studies have found that studies with statistically significant results were more likely to be published than studies with nonsignificant results, and they had significantly shorter times to publication.
Selective or delayed publication of the findings of studies with unwanted results seems to be a widespread but undocumented problem. For example, â€œThe Scientific Consensus on Climate Changeâ€ by Nancy Oreskes found of 913 papers published between 1993 and 2003 with the words â€œglobal climate changeâ€ in their abstracts that â€œNot one of the papers refuted the claim that human activities are affecting Earthâ€™s climateâ€. Although this was widly regarded as evidence of a consensus on global warming, a contribution from selection and publication bias is also highly likely.
In theory, evidence should be believed only if it is produced from well designed studies. Reviews of the medical literature suggest that most studies have variable randomization, small sample sizes and unclear control of bias. As the limitations of studies are frequently not addressed within the study, a systematic review of the evidence is necessary to identify limitations, such as bias or inadequate statistical power. Research on sources of bias is important to empirical fields. All methodological studies could benefit from the influence of better statistical designs.