Cherry Picking — The paper formerly known as “Voodoo Correlations in Social Neuroscience”

Edward Vul, Christine Harris, Piotr Winkielman, & Harold Pashler have published research that provides useful insights into the practice of ‘cherry picking’ or prior selection of desirable results leading to exaggerated significance. They also demonstrates the effect in a comprehensive survey of studies in the field of social neuroscience.

To further ‘pin the thumbs of researchers to the table’, and ensure they are noticed and not ignored, they name all the studies explicitly, listing those that exaggerate significance and those that don’t. This is a great example of how not to win friends while influencing people, and gets 5 stars from me. Here is the back story on the brow-chewing response by his colleagues.

The statistical basis of the paper is this: the strength of the correlation observed between measures A and B (rObsA,ObsB ) reflects not only the strength of the relationship between the traits underlying A and B (rA,B), but also the reliability of the measures of A and B (reliabilityA and reliabilityB). In general,

rObsA,ObsB = rA,B * sqrt(reliabilityA * reliabilityB)

As the maximum rA,B=1 for perfect correlation, the reliabilities of the two measures provide an upper bound on the possible correlation that can be observed between the two measures (Nunnally, 1970).

The problem is that many reported correlations, such as a subject’s proneness to anxiety reactions (Carver and White, 1994 reference omitted from the paper) correlated at a very high r=.96. Measures of personality and emotion evidently do not often have reliabilities greater than .8. Neuroimaging measures seem typically to be reliable at .7 or less. Assuming a perfect correlation the maximum that could be obtained would be sqrt(.8 * .7), or .74. A correlation of 0.94 is therefore impossible.

The result must there fore have been achieved by some process of ‘data-peeking’ or ‘cherry-picking’, thresholding higher correlations in the results while discarding those results that are uncorrelated.

An analogous situation in climate science is the selection of a subset of tree-ring proxies by calibration on global temperatures. This procedure alone can produce a hockey-stick shaped temperature history from random data series. It should be possible to estimate the expected correlation given (1) reliability of tree ring signals for given climate changes, (2) reliability of given climate measurements. The square root of the product of these numbers should set an upper limit on the calibration correlation. If published calibration statistics exceed this figure, the a sample of poorly correlated trees must have been discarded in order to enhance the correlation.

Using this process it should be possible to quantify the degree of ‘cherry-picking’ that has taken place, resolving one of the main contentions that skeptics have had with this field.

HT to Geoff Sherrington.


Nunnally, JC. Introduction to Psychological Measurement. New York: McGraw-Hill; 1970.

March 2009 Global Temperatures from RSS

Its the time of the month for the competition to guess the change in global temperatures for the previous month. Currently Jan Pompe is leading. Luck or insight? Time will tell.

Results from RSS normally are released on the 5th of the month. Voting is now open for March. Place your vote below. All with at least one correct guess will be listed.

Update: RSS is out, and down.

Continue reading

Green Technology Mobsters

Jeremy Clarkson on the reaction to his review of the Tesla Roadster.

I fear that what we are seeing here is much the same thing professors see when they claim there is no such thing as man-made global warming. Immediately, they are drowned out by an unseen mob, and then their funding dries up. It’s actually quite frightening.

The problem is, though, that really and honestly, the US-made Tesla works only at dinner parties. Tell someone you have one and in minutes you will be having sex. But as a device for moving you and your things around, it is about as much use as a bag of muddy spinach.

Redoubt now Ultraplinian

The rumbling Alaskan volcano Redoubt has exploded producing a stratosphere-reaching plume in excess of 60,000 ft (17 km). An eruption is termed ‘ultraplinian’ if its ejecta reaches the stratosphere, about 10km in height. Dust and gases in the stratosphere are known to depress the global temperature for up to a few years after the eruption. The extent of cooling depends on the amount and type of material, the size and duration of ultraplinian eruption, and the latitude (high latitude eruptions like Redoubt are less effective than lower ones).

The plume could be seen easily on infrared here (top center of first radar).

Continue reading

Nir Shaviv explains climate sensitivity

Transcript of the introduction of talk by Nir Shaviv, skeptical Astrophysicist, at the 2009 Heartland Conference entitled: New solar climate links and their implications to our understanding of climate change (listen to audio).

In the introduction, Nir explains the concept of climate sensitivity very clearly. After this, one can understand the origin of such claims as ‘if the hockey stick is wrong, then sensitivity is higher than we thought and global warming will be worse’.

OK I am glad to see the number of people who have evaporated is not very large in this last 5 min break.

To answer a question that was asked before I think one of the problems with the models, first of all with respect to feedback, it is cloud cover. That because you cannot resolve very small scales the cloud cover physics is basically parameterized with a recipe, and because its parameterized with a recipe, whichever recipe you decide to use, whatever you cook depends on the recipe that you use.

Continue reading

Global Temperature Change and Geomagnetic Field Intensity

Alan Cheetham drew my attention to a post on his blog, showing the close relationship between geomagnetic field strength, and rate of temperature change (warming in the N Hemisphere and cooling in the S Hemisphere). The idea is that the the effect of cosmic rays on the Earth’s temperature by seeding low clouds, will be most apparent where the magnetic field is weakest. Maps of the geomagnetic field show an uncanny correlation with ‘recent warming’ (UAH 1978-2006):

Continue reading

Weird Equations

This is a brilliant riff on the financial crisis and how it got to where it is today.

Some of the things you learn in an advanced maths degree, are that 1+1 is not always 2 (as in modulo 2 with addition), and big numbers can be small numbers (like the measure of a Cantor set). Irrelevant, but by way of introduction, small numbers like the size of executive bonuses at AIG relative to the scale of the company bailout, can be large numbers when it comes to their impact, as shown by the panicky half-baked legislation passing into US law.

Here are some selected paragraphs from the Rolling Stone article that really put the bat to the numbers.

Nor did anyone mention that when AIG finally got up from its seat at the Wall Street casino, broke and busted in the afterdawn light, it owed money all over town — and that a huge chunk of your taxpayer dollars in this particular bailout scam will be going to pay off the other high rollers at its table. Or that this was a casino unique among all casinos, one where middle-class taxpayers cover the bets of billionaires.

The mistake most people make in looking at the financial crisis is thinking of it in terms of money, a habit that might lead you to look at the unfolding mess as a huge bonus-killing downer for the Wall Street class. But if you look at it in purely Machiavellian terms, what you see is a colossal power grab that threatens to turn the federal government into a kind of giant Enron — a huge, impenetrable black box filled with self-dealing insiders whose scheme is the securing of individual profits at the expense of an ocean of unwitting involuntary shareholders, previously known as taxpayers.

The problem was, none of this was based on reality. “The banks knew they were selling crap,” says a London-based trader from one of the bailed-out companies. To get AAA ratings, the CDOs relied not on their actual underlying assets but on crazy mathematical formulas that the banks cooked up to make the investments look safer than they really were. “They had some back room somewhere where a bunch of Indian guys who’d been doing nothing but math for God knows how many years would come up with some kind of model saying that this or that combination of debtors would only default once every 10,000 years,” says one young trader who sold CDOs for a major investment bank. “It was nuts.”

The following February, when AIG posted $11.5 billion in annual losses, it announced the resignation of Cassano as head of AIGFP, saying an auditor had found a “material weakness” in the CDS portfolio. But amazingly, the company not only allowed Cassano to keep $34 million in bonuses, it kept him on as a consultant for $1 million a month. In fact, Cassano remained on the payroll and kept collecting his monthly million through the end of September 2008, even after taxpayers had been forced to hand AIG $85 billion to patch up his fuck-ups. When asked in October why the company still retained Cassano at his $1 million-a-month rate despite his role in the probable downfall of Western civilization, CEO Martin Sullivan told Congress with a straight face that AIG wanted to “retain the 20-year knowledge that Mr. Cassano had.” (Cassano, who is apparently hiding out in his lavish town house near Harrods in London, could not be reached for comment.)

Then, in January 2009, the company did it again. After all those years letting Cassano run wild, and after already getting caught paying out insane bonuses while on the public till, AIG decided to pay out another $450 million in bonuses. And to whom? To the 400 or so employees in Cassano’s old unit, AIGFP, which is due to go out of business shortly! Yes, that’s right, an average of $1.1 million in taxpayer-backed money apiece, to the very people who spent the past decade or so punching a hole in the fabric of the universe!

The bonuses are a nice comic touch highlighting one of the more outrageous tangents of the bailout age, namely the fact that, even with the planet in flames, some members of the Wall Street class can’t even get used to the tragedy of having to fly coach. “These people need their trips to Baja, their spa treatments, their hand jobs,” says an official involved in the AIG bailout, a serious look on his face, apparently not even half-kidding. “They don’t function well without them.”

While the rest of America, and most of Congress, have been bugging out about the $700 billion bailout program called TARP, all of these newly created organisms in the Federal Reserve zoo have quietly been pumping not billions but trillions of dollars into the hands of private companies (at least $3 trillion so far in loans, with as much as $5.7 trillion more in guarantees of private investments). Although this technically isn’t taxpayer money, it still affects taxpayers directly, because the activities of the Fed impact the economy as a whole. And this new, secretive activity by the Fed completely eclipses the TARP program in terms of its influence on the economy.

None other than disgraced senator Ted Stevens was the poor sap who made the unpleasant discovery that if Congress didn’t like the Fed handing trillions of dollars to banks without any oversight, Congress could apparently go fuck itself — or so said the law. When Stevens asked the GAO about what authority Congress has to monitor the Fed, he got back a letter citing an obscure statute that nobody had ever heard of before: the Accounting and Auditing Act of 1950. The relevant section, 31 USC 714(b), dictated that congressional audits of the Federal Reserve may not include “deliberations, decisions and actions on monetary policy matters.” The exemption, as Foss notes, “basically includes everything.” According to the law, in other words, the Fed simply cannot be audited by Congress. Or by anyone else, for that matter.

For the rest of 2008, the numbers (in the weekly H4 reports) remained similarly in the stratosphere, the Fed pumping as much as $125 billion of these short-term loans into the economy — until suddenly, at the start of this year, the number drops to nothing. Zero.

The reason the number has dropped to nothing is that the Fed had simply stopped using relatively transparent devices like repurchase agreements to pump its money into the hands of private companies. By early 2009, a whole series of new government operations had been invented to inject cash into the economy, most all of them completely secretive and with names you’ve never heard of. There is the Term Auction Facility, the Term Securities Lending Facility, the Primary Dealer Credit Facility, the Commercial Paper Funding Facility and a monster called the Asset-Backed Commercial Paper Money Market Mutual Fund Liquidity Facility (boasting the chat-room horror-show acronym ABCPMMMFLF). For good measure, there’s also something called a Money Market Investor Funding Facility, plus three facilities called Maiden Lane I, II and III to aid bailout recipients like Bear Stearns and AIG.

When one considers the comparatively extensive system of congressional checks and balances that goes into the spending of every dollar in the budget via the normal appropriations process, what’s happening in the Fed amounts to something truly revolutionary — a kind of shadow government with a budget many times the size of the normal federal outlay, administered dictatorially by one man, Fed chairman Ben Bernanke. “We spend hours and hours and hours arguing over $10 million amendments on the floor of the Senate, but there has been no discussion about who has been receiving this $3 trillion,” says Sen. Bernie Sanders. “It is beyond comprehension.”

If you are interested in knowing more about Maiden I, II & III, here is a pdf complete with boxes and arrows.