The PhilPapers Fallacy (Part 5 of 9)
Table of contents
Part 2: Relevant expertise matters
Part 4: Selection effects matter
Part 5: How PhilPapers respondents interpreted the survey questions matters
Part 7: Philosophical fashions matter
Part 8: Demographic, social, and cultural forces matter
2.4 How PhilPapers respondents interpreted the survey questions matters
Philosophical questions are notoriously tricky. Yet the PhilPapers survey presents philosophers with fairly blunt, straight forward questions. It also presents participants with dichotomies or limited sets of choices. Even though an “other” option is available, people may be inclined to favor the options provided. My guess is that if you presented more response options explicitly (that is, if participants weren’t able to endorse less common views only by choosing “other” and writing in a response) that this would change the distribution of response. You might also get a different distribution of responses if you changed which response options were presented, or even if the order they are presented in (I am not sure if order of response options was randomized).
For example, consider this question:
“Aesthetic value: objective or subjective?”
43.5% of respondents favored objective, while 40.6% favored subjective. It is reasonable to ask how they interpreted this question. That is, what did philosophers responding to this question take “objective” and “subjective” to mean? If they frequently interpreted the question in substantively different ways from one another, that would mean that the response percentages we are left with reflect endorsement of different interpretations of the question and, critically, may reflect different interpretations than what those who conducted the survey intended to ask. For what it’s worth, I don’t know what exactly they intended to ask with this question.
In their commentary on the design of the survey, they say the following:
We wanted to include a question on aesthetics, but it wasn't clear what to include. We consulted with a number of philosophers working in the area, and there was a consensus that a question along these lines would be best. Of course "objective" and "subjective" can be given various interpretations, but experts preferred this formulation to formulations in terms of aesthetic realism or relativism.
Fair enough. It’s important that they acknowledge that the terms they used “can be given various interpretations.” However, note that they add “but experts preferred this formulation to formulations of aesthetic realism or relativism.” While this terminology may have made more sense for specialists in aesthetics, did that make the question more clear for people who do not specialize in aesthetics? Did it influence how they interpreted the question? I don’t know, and I doubt anyone else does. Note that the response pattern for specialists in aesthetics had an even higher proportion in favor of objectivism, with 58% favoring objectivism and only 24.7% favoring subjectivism.
Two concerns there. First, this question is about metaaesthetics. I’m not sure what proportion of specialists in aesthetics specialize in metaaesthetics, but ethics typically involves independent specializations for metaethics and other areas of ethics. Such listings aren’t available for aesthetics, even though one would find different subspecializations there as well. So the people who chose this specialization may not be specialists for questions like this any more than those who study moral philosophy in general are specialists in metaethics.
Furthermore, while the sample size wasn’t small, it wasn’t large, either: 93 specialists responded to this question.
Finally, this is a prime example of where selection effects may come into play. Should we expect on average that people who pursue the study of aesthetics would be more, less, or equally likely than philosophers who specialize in other areas, or nonphilosophers, to endorse aesthetic objectivism? My suspicion is that the answer is “more likely,” and that the reason we see a higher rate of objectivism for specialists is that people disposed towards objectivism are more likely to specialize in this subfield.