Sorry, Wrong Number

Andrew Gelman

Sorry, Wrong Number
For some pundits, statistics can be an inconvenient truth. (William B. Plowman/NBC/NBC NewsWire via Getty Images)

How do bad numbers get into circulation in our political discourse, and how do they stay there, even after being refuted?

As a statistician and political scientist, I care about getting the numbers right, and I am also interested in how people get things wrong. With economic statistics, it is often all about interpretation: were President Obama’s policies a failure given that unemployment was higher at the end of his first term than when he took office, or were his policies a success given that the unemployment rate was in decline? Are record-high budget deficits a national scandal or at a reasonable level as percentage of GDP? But there are also examples of hard numbers, political statistics that have lodged themselves in outlets such as The New York Times opinion page even though they are just plain wrong. What can be done to correct this sort of mistake?

Errors in newspaper op-eds fall into a kind of limbo: they are influential and get the attention of millions of readers (including many in the economic and political elites), but they are labelled as opinion pieces and thus are not subject to the same level of caution and fact-checking associated with news articles. The Times and other leading news organizations will gladly run corrections if reporting in a news article is incorrect, but they seem to have a much lower bar for sloppy analyses in the opinion pages.

1. The Happy Tea Partiers

Take Arthur Brooks, conservative scholar and president of the American Enterprise Institute, who has written extensively on happiness. In this particular op-ed, Brooks presents a lot of statistics in a very reasonable-sounding way, in the Brooksian mode of low-key concern:

“Who is happier about life — liberals or conservatives? The answer might seem straightforward. After all, there is an entire academic literature in the social sciences dedicated to showing conservatives as naturally authoritarian, dogmatic, intolerant of ambiguity, fearful of threat and loss, low in self-esteem and uncomfortable with complex modes of thinking. And it was the candidate Barack Obama in 2008 who infamously labelled blue-collar voters “bitter,” as they “cling to guns or religion.” Obviously, liberals must be happier, right?

Wrong. Scholars on both the left and right have studied this question extensively, and have reached a consensus that it is conservatives who possess the happiness edge. … This pattern has persisted for decades. The question isn’t whether this is true, but why.”

Brooks concludes: “None, it seems, are happier than the Tea Partiers, many of whom cling to guns and faith with great tenacity. Which some moderately liberal readers of this newspaper might find quite depressing.”

Where did Brooks’s dramatic and surprising claim that Tea Partiers are the happiest Americans come from? Jay Livingston, a sociologist at Montclair State University in New Jersey, did some detective work. He took a look at General Social Survey data and found that, if you average data from 1972 through 2010, you indeed find self-identified extremely conservative people to be the happiest (on average), among all ideological categories. But if you just look at 2009-10 (i.e., the Tea Party era), self-identified extremely conservative people turn out to be the most unhappy, as these graphs point out.

This mistake is consequential—the data show the reverse of the pattern claimed in the op-ed—but the steps leading to the mistake are understandable. Brooks took a high-quality data set from past data and assumed the pattern he saw would remain valid. He carefully studied the data on happiness and political attitudes several years ago in the course of writing a book, and when writing this recent op-ed he did not bother checking with recent data. That is understandable. We cannot think to check everything.

Still, I did not like the idea of those false Tea Party numbers appearing uncorrected in the newspaper, and I sent the editors a note explaining the situation. The editorial-page team has not, to my knowledge, ever run a correction. I understand that correcting errors is not the top priority of the Times, but as a statistician I remain upset.

2. The Declining Jews

Ron Unz is a former businessman and political activist who posted a long article claiming, among other things, that Harvard University discriminates in favor of Jews in its undergraduate admissions. He based this claim on counts of Jewish-appearing names among Ivy League undergraduates, National Merit Scholar semi-finalists, International Mathematical Olympiad participants, and other lists of high-achieving high-school students. Unz’s claims originally appeared in The American Conservative, an obscure magazine published by Unz and originally associated with Patrick Buchanan, but gained wide circulation after being touted by New York Times columnist David Brooks, who wrote:

“You’re going to want to argue with Unz’s article all the way along, especially for its narrow, math-test-driven view of merit. But it’s potentially ground-shifting. Unz’s other big point is that Jews are vastly overrepresented at elite universities and that Jewish achievement has collapsed. In the 1970s, for example, 40 percent of top scorers in the Math Olympiad had Jewish names. Now 2.5 percent do.”
It turned out, though, that Unz’s numbers were way off. It is not so easy to count Jews based on their names. Unz’s estimates came from different, incompatible sources, and he used different rules when looking at different lists. The purported drop from 40% to 2.5% is actually a much more gradual decline from 25%-30% to 12%-15%, easily explainable based on demographic changes and increased competition from Asian-Americans in recent decades.

I learned of these problems from Janet Mertz, a professor of oncology at the University of Wisconsin who has published some articles in recent years on the sex and ethnicity distribution of high-end mathematics achievement. After reading David Brooks’s column a few months ago, Mertz tried with no success to correct the record. Brooks and The New York Times did not respond to her emails, did not publish any of her letters, and Unz reports that he and others “ignored or dismissed” Mertz. I looked into it and found Mertz’s criticisms (along with others sent to me by another correspondent) to be convincing. Indeed, Unz later admitted that he derived his 40% and 2.5% calculations from “five minutes of cursory surname analysis.” There has still been no correction in Brooks’ column or elsewhere in the Times.

One could argue that an opinion writer has more latitude because the reader knows that the op-ed page is for, well, opinion, and not objective reporting. But columnists do issue corrections on their own when they see fit, including Brooks. In one recent example, he ran the following correction of one of his columns:

“An earlier version of this column misstated the location of a statue in Washington that depicts a rambunctious horse being reined in by a muscular man. The sculpture, Michael Lantz’s ‘Man Controlling Trade’ (1942), is outside the Federal Trade Commission, not the Department of Labor.”

I was amazed to see that the Times considers the location of a statue to be worthy of a formal correction, while an erroneous published number on Jewish academic achievement, a number that is off by a factor of five, is allowed to stand.

3. How Did It All Go Wrong, and How Can We Do Better Next Time?

It should be no surprise that newspaper columnists writing on deadline make mistakes, or that a political activist can get hold of a number and refuse to let go, no matter how carefully his error is pointed out to him. What is scarier to me is how this behavior mirrors similar practices in science, and the way we learn, more generally.

In statistics, it is typically necessary and appropriate to combine information from different sources. But when doing so, one should calibrate: where possible, one should check that different data sources and methods give similar answers when estimating the same quantity. That is one thing that was not done in the examples above.

How do we think about this from a statistical perspective? A typical example of one of these stories starts with a jolt, with data that at first seems surprising but then can be fit into one’s larger worldview. In the case of Arthur Brooks, the surprise was survey data revealing extreme conservatives to be happier than other Americans. This is not what one would expect given the “angry” reputation of Tea Partiers, but, upon reflection, it is consistent with Brooks’s view of conservatives as generous, well-adjusted people, expressed in his books such as Who Really Cares: The Surprising Truth about Compassionate Conservatism and Gross National Happiness.

As for Unz, he has long been writing about the high proportion of Jews and Asians at Harvard, but it is perhaps only recently that he thought to get some lists of Mathematical Olympiad participants. He took a glance through, tried to count the Jewish names, and found a decline from 44% to 2.5%—a drop by a factor of 17. One’s first reaction is: That’s big news! And the second reaction could echo this initial take: even if the data are not perfect, this might seem too big a discrepancy to have occurred by mistake. In fact, though, large and dramatic numbers can be wrong, and in this case it turned out that a series of mistakes was enough to invalidate the larger claims. To track the chain of errors required some work, but once that work had been done, the published numbers were clearly wrong.

For David Brooks, the big surprise was seeing something interesting in a borderline anti-Semitic article in a fringe publication. But the article appeared to be backed up by hard numbers. The results of the apparent evidence made it that much more convincing. Perhaps Unz’s very outrageousness serves as a sort of protective coloration; his conclusions are so disturbing that we want to be careful to not dismiss him out of hand. So when he presents numbers implying that “Jewish achievement has collapsed,” we want to be fair to him and take his argument seriously. This is the essence of what Brooks was writing in his column: we should not dismiss Unz’s arguments just because the conclusions may be unpalatable. But in this case his numbers did turn out to be wrong.

As the saying goes, just because something is counterintuitive, that does not make it true. Mistakes are inevitable. But when a published number is clearly wrong, and the publication is informed of the error, and the number is still not corrected, it sends all the wrong messages and gives all the wrong incentives if the goal is clear, honest, and accurate communication. It encourages all sides in any given debate to use data carelessly if they have reason to believe that a wrong number, when injected into public discussion, will not be corrected.

Share This
About The Author:

Andrew Gelman is professor of statistics and political science and director of the Applied Statistics Center at Columbia University. You can follow him on his blog as well as on The Monkey Cage.

  • highly_adequate

    One point you seem to be missing is the point of statistics anyway.

    Yes, Unz appears to have gotten the numbers very far wrong with the decline in Jewish representation at the high end of mathematical achievement at the student level. But even if the multiple by which it declined is not 17, but rather 2, how much is his overarching point affected? If Asians, by your own admission, have greatly increased their representation at these highest levels of mathematical achievement, and that of Jews has declined by a factor of 2 (only small compared to a number like 17), why is it that the number of Jews in elite universities like Harvard has only seemed to go up over that period, and the number of Asians seems to be fixed at around 15%?

    To dismiss,as you do, the new numbers as if the decline in Jewish achievement is no big deal is to refuse to connect those numbers with the actual questions motivating the investigation in the first place. And that, unfortunately, is just too typical of how statisticians behave — never, it seems, seeing the larger picture.

    • http://www.gwern.net/ gwern

      > But even if the multiple by which it declined is not 17, but rather 2, how much is his overarching point affected?

      Sizes matter, and his point is affected massively, because the smaller the decline, the more likely it’s explained by any of the possible small methodological details and quibbling one could make. A factor of 17 would be hard to explain by suggesting Unz miscounted, but a factor of 2 could easily be explained by something like intermarriage or variation from year to year. (An analogy: you run a business and one night your cashier reports that $0.05 is missing from their register; you shrug and write it off as probably due to sales tax rounding or something; the next night, they report $500 is missing; you call the police and fire them.)

  • mike rappeport

    A critical point on any use of Jewish surnames as an indicator of number of Jews is the dramatic change in the intermarriage rates among Jews which began with the generation of the parents of today’s college students. Based on an intermarriage rate which exceeds 50% of marriages, and data which indicates roughly the same number of children of the non-ultra Orthodox (few of whom go to college) intermarried as of children of the in-married consider themselves Jewish, All together perhaps 1/4 to 30% of the kinds off Jews who go to elite colleges have fathers who were neither born nor raised Jewish and thus surely were very unlikely to pass on a Jewish surname. In the current college generation, this alone accounts for a factor of around .7 in the number of those who consider themselves Jewish but do not have a “Jewish name”. Anyone who can write an article involving Jewish surnames who doesn’t prominently mention this either doesn’t know the community or has an ax to grind.

    • highly_adequate

      You are missing an important point about how a surname analysis reduces the problem you’re talking about.

      Yes, there may be today many more Jews — actually, partial Jews of course — who don’t have a clearly Jewish name because of intermarriage. But there are presumably going to be effectively as many individuals with a clearly Jewish name who are not full Jews, but who will be counted as Jews for the analysis. The issue of intermarriage presents us with the problem of how we count Jews. Do we include only full Jews? Do we count partial Jews, but only count the fraction that’s Jewish, or do we count any partial Jew as one Jew?

      Rather than get into the various approaches to handling intermarriage, let me just make the quick observation that Jewish relative achievement should be captured by the ratio of percentage of high achieving Jews in the population to the percentage of Jews in the population. You may be right that the surname analysis falls short by a factor of .7 in counting the percentage of high achieving Jews (by which I assume you mean that any partial Jew counts as one Jew and that the percentage of high achieving Jews in the population thus defined is actually 1.7 times the percentage produced by surname analysis). Yet you neglect to note that it also would fall short by a factor of .7 in counting the percentage of Jews in the larger population. What’s good for the numerator goose has to be good for the denominator gander. These factors cancel out, of course, leaving the surname analysis as a satisfactory approach. A rise or fall in the ratio thus measured should have consequences in terms of representation in elite institutions, which, in a meritocratic society, should reflect relative achievement.

      Even if we don’t rely on our somewhat unreliable sense of what a Jewish name might be, but focus only on names that pretty much everyone concedes is Jewish — e.g. Goldstein, Cohen, Levy — then a decline in the appearance of those names in lists of high achievers would seem to signal a comparable decline among all Jews, no matter their names, Why should it be that only the Goldsteins, Cohens, and Levys decline? Why should they not be representative of Jews as a whole? (Of course, there is the statistical issue that the smaller the number of Jews who might fall under a set of Jewish names, the larger the error bars in any inference.)

      If it indeed remains true that under consistent surname analysis the Jewish representation at, say, Harvard is 2 times as high as Jewish achievement, then I should think an explanation is in order.

      • NB

        Using consistent surname analysis (Weyl Analysis), one finds that 6-7% of current Harvard undergraduates are Jewish, which is the same result one obtains for the % of Jewish NMS semifinalists using Weyl Analysis.

        I am Prof. Gelman’s correspondent who performed Weyl Analysis on the Harvard alumni directory. Unz’s description of Weyl Analysis was ambiguous, and when I first performed Weyl Analysis on the Harvard alumni directory, I incorrectly assumed that Unz meant Gold* when he wrote “Gold—”. In order to figure out how Unz actually performed Weyl Analysis, I had to reproduce his finding that Jews represent 6-7% of NMS semifinalists according to Weyl Analysis. Using the same methodology on the Harvard alumni directory that enabled me to reproduce Unz’s NMS results, I obtained the estimate that Jews represented 7-9% of Harvard College students in Fall 2008 (as opposed to the higher estimate that Prof. Gelman initially reported). I’ve also performed Weyl Analysis on the current Harvard College directory, which gave the estimate that 6-7% of current Harvard undergraduates are Jewish. Thus, there is no discrepancy when you use the same methodology on both data sets. (Since more than 6-7% of Harvard students are Jewish, it is evident that Weyl Analysis yields underestimates.)

  • Evil_Spock

    I was initially very interested in the Unz article and therefore grateful at the work Janet Mertz did on this – she convinced me that Unz was terribly wrong. But I disagree with its characterization as “a borderline anti-Semitic article in a fringe publication.” Let’s just call it “wrong” rather than anti-Semitic, borderline or not. If it had been right, it would have been interesting, and (I’d hope) especially of interest to Jews. If this were happening – Jews were getting huge bonuses in college admission – I would want to know about it, I would object to it, and I would be appreciative of whoever pointed it out.

    Getting publications to make statistical corrections, even in news articles, is just impossible. There’s still an Atlantic article that says “1 out of 2 bachelor’s degree holders under 25 were jobless or unemployed”, despite my best efforts to get them to correct it. (It’s actually “underemployed,” for a certain definition of underemployed.) There’s also an AP article with a graphic reporting that “9 in 10 female marines experienced unwanted sexual contact” (it’s actually 9 in 10 of the the less than 10% who reported any kind of unwanted attention). Both of these numbers should have made the author double- and triple-check: they’re stunning, if you have have any sense of context about these issues. Which they clearly didn’t.

  • Kevin

    Andy, Occam’s Razor says that this article is way too long and too clever…… you should just blame everything on David Brooks!!!

  • Mike Spagat

    Everything that Andrew says (in this particular article) is true and I have no doubt that if Andrew had said something wrong he would have corrected himself already.

    However, sadly, I fear the problem runs still deeper than he indicates. Even if the NYT’s, or the two Brook’s themselves, had issued corrections people would still be citing the false numbers they have circulated. Nobody seems to notice retractions.

    I experienced this phenomenon first hand a few weeks ago. I wrote a magazine article called “The Iraq Sanctions Myth” in which I devoted some space to an old survey that that had been formally retracted after having injected a false number into the public domain. Eventually some guy showed up in the comments section and cited the retracted survey as refuting my article.

    Perhaps in a few days someone will log on here and cite David Brook’s article to refute Andrew’s. After all how can Tea Partiers be the least happy group in America (as Andrew falsely claims) when David Brooks writes that in reality they are the most happy? I don’t see why the NYT’s should issue a retraction.

  • LarryRothfield

    Arthur Brooks has a lot to answer for. Here’s another instance of his bias in marshalling statistics: In his book on charity the central claim was that conservatives are more charitable than liberals. I remember being puzzled at what this meant and asked him directly whether his economic analysis of charity included the income forgone by those who decide to work in the non-profit sector. To me, that seemed the obvious way in which most liberals express their interest in helping others: the doctor who chooses to work at a public hospital rather than go into private practice, the attorney who forgoes corporate law for a career serving poor people, etc., are all giving, and in a much more profound way than tossing money into the collection pot or volunteering on weekends. Surely that giving of one’s life ought to be recognized as showing charity (in the sense of virtuous sacrifice for the sake of others). And surely it could be measured by looking at the income gaps for similar jobs between for-profit and non-profit sectors. Brooks admitted he had not included foregone income, and said he had never thought of doing so. Quite an oversight, especially from a former cultural economist, a field in which conservatives rightly remind liberals all the time that the government gives artists more support indirectly (through tax exemptions) than directly (through subsidies). But if you only count the giving that the side you like does, it is a lot easier to arrive at the answer you want.

    I haven’t seen him making any effort to reel his book’s central claim back in since we talked, of course.

  • Justin

    “As for Unz, he has long been writing about the high proportion of Jews and Asians at Harvard…”

    I don’t think Unz claimed Asians are over represented relative to their academic performance. Yes, high proportion of Asians relative to percentage of whole population but as a whole I felt Unz was saying Asian enrollment has been suppressed at Ivies. To counterbalance this claim he does his quasi-analysis of Jews, using their supposed “over representation” as a foil. This analysis was flawed, of course, but I think the general comparison between Asians and Jews was accurate, no?

    I also find it strange how 99% of the discussion about Unz’s article focused on the Jewish portion and seemed to treat the questions on Asian representation as a non-issue.

    Finally, I would hesitate to call the American Conservative an “obscure magazine.” Why don’t we wait and see how Symposium’s doing (in terms of popularity) after year? Content in this first issue is great, but I fear Symposium may be equally “obscure” as AmCon.