Other studies find that judges' votes are correlated with political views. Indeed, even in the same case judges from across the aisle vote in different ways. Those are observational studies, so suffer from issues with internal validity. Our study has greater internal validity because it is randomized, but suffers from external validity. We study students not judges. We don't provide as much context or room for reasoning. Yet our study finds similar results. When there is evidence of political influence on one side and the other, it takes a great deal of faith to suppose that judges are fully insulated from their own political preferences. The commentators provide no evidence on which to base that faith.
Our remaining claim is that the ordinary readers framing helps mitigate the effect of preferences. Unfortunately, there is no observation data on judges corroborating our findings here. Perhaps the data from Brudney and Ditslear would help. In any case, our data suggest that the ordinary readers framing could have a potential debiasing effect. This should be explored in future work. Ideal would be a randomized study of judges.
And again, judges are not the only individuals who bear responsibility for interpreting statutes. Indeed, I would conjecture that only a small fraction of behavior is influenced by how judges read a statute. Lawyers working for clients and the government must read statutes and tell clients or government employees how they must behave. Prosecutors must read statutes and decide which individuals to prosecute. All without judicial guidance. In some cases these lawyers will use outside information, such as legislative history. But in many cases they will base their decisions on the text of a statute and a quick judgment. Our study may be more predictive of what influences lawyers in this context.
One final point. Perhaps it would be easier to absorb our findings if the experiment were viewed not as an attempt to see how judges vote but rather as a cognitive psychology experiment. Social scientists have run numerous experiments on students revealing that they have trouble dealing with low probability events, or cannot understand fractions as well as odds, or are averse to ambiguity, or suffer endowment effects. Numerous legal scholars have cited these studies as suggesting that judges may suffer similar foibles. (And judges have surely wondered whether these behavioral findings apply to litigants before them.) It can always be objected that the researcher did not study the target population to which scholars want to extrapolate their findings. But the response is that the studies shed light on psychology and this psychology is not specific to one profession or another.
From this perspective, perhaps it would have been better if, instead of showing students statutory text, we had simply showed them some other writing, such as a newspaper story, and asked them how they interpret that story. I predict we would have found that preferences affect how people read the story. But surely the fact that we actually used statutory text should not make such a cognitive psychology experiment less persuasive. On the contrary, we show how the problem of separating preferences from judgments can affect a person's reading of the same sorts of texts that judges read. We don't claim that judges make their ultimate decisions about cases in the same way our respondents answered their survey questions; but we think it's plausible that judges struggle with the same basic problem our respondents did, and with the same lack of success, at the stage of a case when they are trying to make the same kinds of judgments that our respondents were.