Am I Right? Am I Wrong?

It’s December 1, and I’m still thinking about the political discussion in our house on Thanksgiving, which I mentioned here. It wasn’t the first time I’d been invited inside the mind of a true believer, but because the argument stayed polite and level-headed, it hit me harder: When someone believes in a fixed truth with an unwavering and lifelong conviction, then everything that happens anywhere in the world serves to reinforce that truth. He only pays attention to people and pundits who tell him exactly what he wants to hear. Anything that contradicts his world view is the other side’s propaganda or disinformation.

It stands to reason, then, that people who refuse to consider most of the available information in the world are going to be wrong most of the time. Louis Menard explores that question in this week’s New Yorker:

Prediction is one of the pleasures of life. Conversation would wither without it. “It won’t last. She’ll dump him in a month.” If you’re wrong, no one will call you on it, because being right or wrong isn’t really the point. The point is that you think he’s not worthy of her, and the prediction is just a way of enhancing your judgment with a pleasant prevision of doom. Unless you’re putting money on it, nothing is at stake except your reputation for wisdom in matters of the heart. If a month goes by and they’re still together, the deadline can be extended without penalty. “She’ll leave him, trust me. It’s only a matter of time.” They get married: “Funny things happen. You never know.” You still weren’t wrong. Either the marriage is a bad one — you erred in the right direction — or you got beaten by a low-probability outcome.

It is the somewhat gratifying lesson of Philip Tetlock’s new book, Expert Political Judgment: How Good Is It? How Can We Know? (Princeton; $35), that people who make prediction their business — people who appear as experts on television, get quoted in newspaper articles, advise governments and businesses, and participate in punditry roundtables — are no better than the rest of us.

When they’re wrong, they’re rarely held accountable, and they rarely admit it, either. They insist that they were just off on timing, or blindsided by an improbable event, or almost right, or wrong for the right reasons. They have the same repertoire of self-justifications that everyone has, and are no more inclined than anyone else to revise their beliefs about the way the world works, or ought to work, just because they made a mistake. No one is paying you for your gratuitous opinions about other people, but the experts are being paid, and Tetlock claims that the better known and more frequently quoted they are, the less reliable their guesses about the future are likely to be.

The accuracy of an expert’s predictions actually has an inverse relationship to his or her self-confidence, renown, and, beyond a certain point, depth of knowledge. People who follow current events by reading the papers and newsmagazines regularly can guess what is likely to happen about as accurately as the specialists whom the papers quote. Our system of expertise is completely inside out: it rewards bad judgments over good ones.

The weirdest thing is, psychologists have known for a long time that experts are more likely to be wrong than non-experts, even if it’s news to political junkies:

Expert Political Judgment is just one of more than a hundred studies that have pitted experts against statistical or actuarial formulas, and in almost all of those studies the people either do no better than the formulas or do worse. In one study, college counsellors were given information about a group of high-school students and asked to predict their freshman grades in college. The counsellors had access to test scores, grades, the results of personality and vocational tests, and personal statements from the students, whom they were also permitted to interview. Predictions that were produced by a formula using just test scores and grades were more accurate.

There are also many studies showing that expertise and experience do not make someone a better reader of the evidence. In one, data from a test used to diagnose brain damage were given to a group of clinical psychologists and their secretaries. The psychologists’ diagnoses were no better than the secretaries’.

The experts’ trouble in Tetlock’s study is exactly the trouble that all human beings have: we fall in love with our hunches, and we really, really hate to be wrong.

Test-driving the Louminations

Self-assessment time: When I go out on a limb and make predictions, how often am I right?

My worst call was the 2004 election — I thought Kerry would win, as I noted here. A year later, I think I understand why 60 million people voted for Bush, even though much of what we now know about his presidency could’ve been known then.

I was right about steroids in baseball, since a bit of knowledge in both areas put me ahead of the curve. When ballplayers were doing things that no previous ballplayers had done, and doing them at ages when ballplayers traditionally are well past their peak abilities, there has to be a reason. Steroids explained these freakish events better than anything else. (And, in many cases, ballplayers made it easy by developing freakish physiques to go with their freakish accomplishments.)

In my core area of expertise — exercise and nutrition — I admit to being biased toward strength training and higher-protein, lower-carbohydrate diets. (It would be hard to write books like these without some kind of bias.)

But my overall feeling about those subjects is a bit more nuanced than “I’m right and everyone who disagrees is wrong.” I think all exercise systems and diet plans get people to more or less the same place: That is, if they consciously modify what they’re doing — if they think about food before they eat it and deliberately set out to get “exercise,” whatever they think that means — they get similar health benefits. They’re probably going to weigh less, live longer, and feel better about themselves.

As I’ve written somewhere, the guy who runs marathons isn’t going to look like a weightlifter, and the guy who lifts heavy weights isn’t going to look like a yogi, but chances are the basic health benefits will be similar. I think we have enough data now to suggest that the most extreme plans (the super-low-carb or super-low-fat diets; the six-days-a-week workout plans) will have the most dropouts, and the most flexible and user-friendly plans will have the highest adherence. The fastest and most dramatic results will probably come from the extremes, but I’ve always said that the best plan is the one you can stick with.

If I were to predict the future of my pursuits, I’d say that it’s not good. There’s already evidence that young adults aren’t exercising as much as middle-aged and older people. It doesn’t mean that everyone will stop exercising tomorrow (the health-club industry is still growing), but it does suggest that a smaller percentage of our population will care as much about health and fitness in the future.

And, believe me, I’d love to be wrong about that one.

Tags: Tags: Politics