A lot of studies employ some experimental set up in which participants are prompted to express a view about some topic. Researchers then take these responses to be evidence of what people in general think.
This is strange. How can we distinguish instances in which participants are expressing what they already thought prior to participating in the study from instances in which participants form a view by participating in the study?
For instance, suppose we wanted to test a new product. If we have people try the product, and 80% approve of it, does this mean 80% of the public approves of the product?
Of course not: it's a new product! They've never tried it. We might conclude that *if* they tried it, then 80% of people would approve of it, but that's not the same thing as saying that 80% in fact do approve the product.
Just the same, when philosophers conduct empirical studies on how nonphilosophers think about philosophical questions, and they present participants with a forced choice between two or more philosophical positions that are recognized in the philosophical literature, how can we tell if the responses participants provide reflect views they held prior to participating in the study?
There are at least three other possibilities:
(1) Forced choice
They chose one of the positions because they didn't have any viable alternative. If you give someone the options (a) and (b), most people are going to pick (a) or (b) rather than quit the study or skip the question because this is the easiest thing to do and people generally don’t want to be noncompliant (and if you’re conducting an online study, a person may worry they won’t get paid if they don’t answer).
As a result, a substantial number of people may choose response options that don’t even reflect what they think.
(2) Unintended interpretations
They choose a response that they believe represents their thoughts, but their interpretation of the stimuli or the response option (or both) doesn’t reflect researcher intent. If I ask people if they like “going to the bank,” and they think I’m asking about river banks, and say “yes,” but I intended for them to think of financial banks, I will misinterpret their response, resulting in an invalid measure.
(3) Spontaneous theorizing
They will think about the question for the first time and form a belief. This may or may not reflect how people in general are disposed to think. But there are two important issues:
First, it doesn’t reflect what other people actually believe or think (unless you think dispositions entail beliefs; I don’t know if psychologists typically want to theorize on the basis of questionable philosophical positions).
Second, people don’t simply have a categorical disposition to believe something. The degree to which someone is disposed to endorse something will depend, in part, on the context in which they are prompted to do so. A compelling argument for a view is more likely to dispose someone to endorse the view than a less convincing argument. Other factors may also contribute to a person’s disposition to endorse a view (assumptions about its implications, trust in the person presenting the information to them, clarity of information, salience of alternatives, and so on).
Simply because people prompted in one study to consider whether they endorse X or Y finds that 80% endorse X doesn’t mean that the general public has a disposition to endorse X 80% of the time. At best, this would show that they have a disposition to endorse X in the experimental context that was tested. But not all experimental contexts are going to yield the same 80% response rate.
The “true response rate” may be a myth, since dispositions are sensitive to context, but to the extent that something like a fairly stable disposition across contexts might be obtained (and I think it could be), it would require convergence of a variety of plausible contexts in which a person could be disposed to endorse a given view. Such convergence isn’t attained by conducting a single study. A single study is almost never enough to evaluate dispositions. This is no idle speculation: studies on whether ordinary people endorse compatibilism about free will yield different outcomes, as do studies on whether ordinary people endorse moral realism. While this variation must be due in part to differences in the samples, variations in wording and other factors relevant to a study design plausibly influence response patterns.
Researchers typically are or at least should be aware of this, and yet both researchers and laypeople reading their studies routinely overgeneralize from a single study or paradigm.
These are difficult problems, and I think they pose a distinctive challenge to experimental philosophy.
I suspect many experimental philosophers have operated under the assumption that ordinary people already have philosophical intuitions, and the task of their studies is to discover what those intuitions are. I worry that they have not seriously grappled with the possibility that the very studies they conduct are manufacturing philosophical positions, not discovering them.
I wonder if these observations have any implications for the viability of longitudinal work in XPhi? On one hand, if people are hyper sensitive to the experimental context, it may be difficult to say whether changes in response patterns over time are due to changes in the underlying disposition. On the other hand, if people were not so sensitive, you might worry that repeated questioning over time was effectively “training” people in philosophical reasoning, such that by the end of the study age/stage of life was confounded by philosophical training.