Building models and using advanced statistics to unpack vexing questions in the social sciences makes for a neat, enlightening, and intellectually stimulating pursuit. It’s how I spend most of my days, happily so. But social science models, both theoretical and empirical, aren’t all cut from the same quality cloth.
Many models as well as novel, compelling, results turn out to be wildly off the mark. Sometimes models are wrong in the details but largely correct in the generalities. Other times, even the general conclusions are suspect. And let’s face it, sometimes the results are actually being fudged. Consider some recent misses in the social sciences, instances when new and novel findings were initially celebrated until other researchers showed that the early results were not entirely correct, or even outright wrong.
- A famous theory from I/O psychology that was influential in the 1960s, Fredrick Herzberg’s Two-Factor Hygiene-Motivation model, has been empirically debunked and largely discarded.
- In economics, a seminal 1994 study by David Card and Alan Kruger looking at the effects of New Jersey increasing its state minimum wage and the consequent impact on fast food industry employment has been convincingly refuted by other labour market economists.
- Also from economics, groundbreaking work that started with Steven Levitt (of Freakonomics fame) seemingly showed that abortion legalization after the 1973 Roe V. Wade ruling led to less crime and better health outcomes among adolescents, due to fewer unplanned or unwanted child births. Though controversial at the time, the result was hailed when it was first published. Since then, it has been vigorously challenged and remains a very debatable contention.
- In political science, Francis Fukuyama’s assertion that international affairs reached an End of History after U.S.S.R fell now belongs in a musty smelling textbook on the history of outdated political theories.
- Back to I/O psychology, there’s the so-called replication crises. A recent and disconcerting twist in that tale was the suspension of Harvard professor Franseca Gino in late-2023.
- In real world forecasting, the Bank of Canada’s yearly forecast for Canada’s economic performance is invariably incorrect, sometimes by just a little and sometimes by a whole lot.
These are just a half dozen examples of social scientists missing the mark. So, what’s going on here? Where did all these errors in seemingly solid conclusions heralded by social scientists come from? Two possible sources are (1) mistakes due to the inappropriate application of statistical procedures, and (2) mistakes due to so-called identification problems.
Mistakes in identification – which basically means that our available data, and our understanding of that data, doesn’t truly jive with the actual population under study – can lead to so-called ‘reflection problems’, where it becomes impossible to disentangle the directional influence between individual and group level behaviors (or more technically, “identification of endogenous social effects”). And there are more technical problems concerning extrapolation from small samples, sparse, censored, and truncated data, misapplication of models when key assumptions don’t hold, and many other inferential challenges. In short, there are many ways to go analytically awry, to put it mildly.
One social scientist closely associated with the above critique is Charles Manski, and it’s worth quoting his words of caution at length. In one influential article, Manski stated that in his view:
“the core problem to be the inherent difficulty of the questions facing the social sciences. The conclusions that one can draw from an empirical analysis are determined by the assumptions and the data that one brings to bear. In social science research, the available data are typically limited and the range of plausible assumptions wide; hence the generally accepted conclusions are necessarily weak. Disagreements about the determinants of human behavior, the nature of social interactions, and the consequences of public policy persist because researchers who analyze the same data under different maintained assumptions reach differently logically valid conclusions” (Masnki, 1993, p. 2 – 3).
I would argue that some of those conclusions do not even comport with solid logic. And sometimes assumptions are little more than wishful thinking, or the creeping in of confirmation biases. The main lesson is this: social science models are heuristics with ingrained assumptions, not the holy utterings of the Lord Almighty. We’d be wise to avoid making assertions of incredible certitude. Analytical humility is a virtue.
- Issac, B. (2024). Harvard Business School Investigation Report Recommended Firing Francesca Gino, The Harvard Crimson (March 14, 2024).
- Manski, C. (1993). Identification problems in the Social Sciences, Sociological Methodology 23: 1 – 56.
- Mounk, Y. (2020). The End of History Revisited, Journal of Democracy 31(1):
- Ropponen, O. (2011). Reconciling the Evidence of Card and Krueger (1994) and Neumark and Wascher (2000), Journal of Applied Econometrics 26(6):
- Wall, T. and Stephenson, G. (1970). Herzberg’s Two-factor Theory of Job Attitudes: a Critical Evaluation and Some Fresh Evidence, Industrial Relations Journal: 41 – 65.
Photo via Silicon Republic
