Can We Really Measure Implicit Bias? Maybe Not

I harbor a moderate preference for white faces. You probably do, too: About 70 percent of people who take the race version of the Implicit Association Test show the same tendency — that is, they prefer faces with typically European-American features over those with African-American features. Since it first went online in 1998, millions have visited Harvard’s Project Implicit website, and the results have been cited in thousands of peer-reviewed papers. No other measure has been as influential in the conversation about unconscious bias.

That influence extends well beyond the academy. The findings come up often in discussions of police shootings of black men, and the concept of implicit bias circulated widely after Hillary Clinton mentioned it during the presidential campaign. The test provides scientific grounding for the idea that unacknowledged prejudice often lurks just below society’s surface. “When we relax our active efforts to be egalitarian, our implicit biases can lead to discriminatory behavior,” according to the Project Implicit website, “so it is critical to be mindful of this possibility if we want to avoid prejudice and discrimination.”

In other words, beware your inner bigot.

But the link between unconscious bias, as measured by the test, and biased behavior has long been debated among scholars, and a new analysis casts doubt on the supposed connection.

Researchers from the University of Wisconsin at Madison, Harvard, and the University of Virginia examined 499 studies over 20 years involving 80,859 participants that used the IAT and other, similar measures. They discovered two things: One is that the correlation between implicit bias and discriminatory behavior appears weaker than previously thought. They also conclude that there is very little evidence that changes in implicit bias have anything to do with changes in a person’s behavior. These findings, they write, “produce a challenge for this area of research.”

That’s putting it mildly. “When you actually look at the evidence we collected, there’s not necessarily strong evidence for the conclusions people have drawn,” says Patrick Forscher, a co-author of the paper, which is currently under review at Psychological Bulletin. The finding that changes in implicit bias don’t lead to changes in behavior, Forscher says, “should be stunning.”

Hart Blanton was not stunned. For the last decade, Blanton, a professor of psychology at the University of Connecticut, has been arguing that the Implicit Association Test isn’t all it’s cracked up to be. In a 2013 meta-analysis of papers, Blanton and his co-authors declared that, despite its frequent characterization as a window into the unconscious, “the IAT provides little insight into who will discriminate against whom, and provides no more insight than explicit measures of bias.” (By “explicit measures” they mean simply asking people if they are biased against a particular group.)

The test works by measuring how quickly people can, for instance, associate African-American faces with positive words versus European American faces with those same positive words. In one round of the test, you’re instructed to press a particular key if a positive word like “pleasure” or “wonderful” flashes on the screen and to press that same key if a white face appears. Then, in another round, the program will tell you to press the same key for darker faces and positive words. It tracks how many mistakes you make and measures how quickly you press those keys, right down to fractions of a second. The site also offers tests that measure bias against other groups, including obese people, the disabled, and the elderly, though it’s the race results that tend to dominate the discussion.

It generally takes people longer to associate a positive word with an African-American face than a European-American face. What’s uncanny is that the test usually works even on people who, like me, know what’s being measured ahead of time and are doing their best to answer at the same speed so as not to be deemed biased.

But those results, Blanton has been saying in paper after paper, year after year, don’t tell us much, if anything.

For the record, Blanton is a 49-year-old white guy who considers himself a liberal and became a psychologist because of an early interest in social justice. A journalist once referred to him as a “conservative intellectual,” which Blanton jokes is wrong on both counts.

Over coffee recently, he sketched out an analogy in his notebook. He drew a graph illustrating how high IQ scores tend to predict achievement, a claim backed up by reams of data. In contrast, the IAT — a sort of IQ test for bias — doesn’t reveal whether a person will tend to act in a biased manner, nor are the scores on the test consistent over time. It’s possible to be labeled “moderately biased” on your first test and “slightly biased” on the next. And even within those categories the numbers fluctuate in a way that, Blanton contends, undermines the test’s value. “The IAT isn’t even predicting the IAT two weeks later,” Blanton says. “How can a test predict behavior if it can’t even predict itself?”

{snip}

Blanton is not saying there’s no such thing as unconscious bias, nor is he arguing that racial discrimination isn’t a deep and abiding problem in American life (though at least one white-supremacist-friendly website has mentioned his research in an attempt to make that case — illustrating how such discussions can be misconstrued). He just thinks that scientists don’t know how to measure implicit bias with any confidence and that they shouldn’t pretend otherwise. “It is such an important problem that it deserves a stronger science,” he says.

Forscher hopes the discussion will move beyond the long-running, sometimes caustic back-and-forth over the IAT. He wants to focus on understanding the root causes of discrimination in order to combat its pernicious effects. As part of that mission, he’s for several years helped train police officers in Madison about bias.

{snip}

“I see implicit bias as a potential means to an end, something that tells us what to do and some possible remedies for what we see in the world,” Forscher says. “So if there’s little evidence to show that changing implicit bias is a useful way of changing those behaviors, my next question is ‘What should we do?’”