Further thoughts on Goodhart's Law and the problem of journal centrality in philosophy

Photo by Nadya Spetnitskaya on Unsplash

Since hiring and career advancement in philosophy depends a great deal on publication in journals, this incentivizes an emphasis on doing philosophy in the form of essays, and, in particular, the kinds of essays that journals want to publish. This encourages a shift away from every other medium for doing philosophy: discussion, debate, spoken philosophy, recording videos, and other forms of writing: diaries, self-reflections, notes, exchanges of letters, books, blogs, and so on. If it isn’t rewarded in the same way as publishing in top journals, people are simply going to do less of it, and the field as a whole will attract fewer people who have skill and talent for these other means of doing philosophy. By discouraging these other activities, this also discourages everyone from investing a great deal in and developing their ability to contribute in these ways, as well. So we two sources of reduction in the quantity and quality every other medium of philosophy:

1. Selection-based reduction: Fewer people with a comparative advantage in doing philosophy in every way other than publishing papers in top journals will be disincentivized and discouraged from contributing to academic philosophy

2. Skill-based reduction: Those who are committed to remaining in the discipline will have less incentive to produce content in these other mediums, and so they will develop less skill at doing so. As a result, the comparative quality of what we do see. People are also less likely to continue doing things they are not comparatively good at so this will lead to a reduction in output as well.

This probably has very bad consequences. Imagine if all chefs were primarily rewarded based on how well they baked bread. Those chefs with the most talent and interest in baking bread would rise to the top, while those with other interests or talents would be comparatively disadvantaged. This would lead chefs with other skills, talents, and interests to be less successful. Some would leave the culinary world altogether. Some would shift towards baking even if it wasn’t their comparative advantage, leading them to make less good food that isn’t bread. Since they dedicate most of their time to making bread, they’d have less practice and experience making things other than bread, so the quality of their non-bread dishes will be worse than if they were equally rewarded for making non-bread. Some would stick it out and insist on making things other than bread, but they would wallow in anonymity. Fewer people would eat their food, and it would be less prominent and have less of an influence on everyone. The net result is that we’d have a lot of very, very good bread, and much less other kinds of good food in general.

I think the same thing is going on in academic philosophy. Philosophers have defended the use of publication in journals as a tool for quickly assessing competence and potential. But what philosophers are failing to appreciate is that there is a two-way causal pathway between the methods we use to measure people’s output and reward them for it, and what people end up doing.

It’s not like philosophers are just going to do philosophy in whatever way they think is best, that journal publication is just some passive means of assessing them, and the ones who do well at this happen to rise to the top. Rather, insofar as philosophers and philosophy students are aware of how they are assessed, they will face extreme pressure to shift their priorities towards doing whatever it is that will make them successful. The net result is just like the bread scenario: people who’d be great lecturers or bloggers or debaters or orators or syllabi developers or conversationalists or mentors or whatever, but are at a comparative disadvantage when it comes to publishing in the most prestigious journals, will be pushed out of the field or suffer permanently worse job offers, chugging away as adjuncts or at less prestigious universities, where they will inevitably receive fewer accolades, their work will enjoy less attention, they will win fewer grants and awards, and their work will be regarded as less central to the field. Many will simply leave the field and stop contributing, leading to a literature biased in favor of those who write the sorts of things that get published in top journals.

This can lead to a snowball effect where emphasis on academic publishing is self-reinforcing: those who rise to the top of the field through publishing in “top journals” then become the editors and reviewers and department chairs and famous important people that everyone else listens to, and via self-selection and strong reasons to defend the status quo that got them where they were, can cement and defend the practices that put them in those positions, while retaining almost every position of power that a person could, in principle, use to change the situation. This will also result in almost all experts not merely have strong biases and a vested interest in retaining the status quo, but genuinely believing that the established methods are the best methods. Those who do or would think differently simply aren’t there to make their case. This gives the impression that there’s a consensus in favor of the status quo because there is a consensus in favor of the stupid quo. Only that consensus is due to selection effects, attrition, and biases as much or more than it’s due to the status quo actually being the best way to approach things.

This is an awful situation, and I have no idea how to fix it.