I initially intended this to be a note but I figured I’d just post it as a blog post. I dashed it off quickly in one sitting and it isn’t intended to reflect the quality of a longer post that I’ve had more time to edit or support with research. I hope to say more about just how absurd and indefensible the current institutions and people’s attitude towards them are, but it’ll be a while before I have the time to do that.
Note: Unless otherwise specified, when I refer to “peer review” I’m typically referring not to the general idea of using relevant experts to evaluate one another’s work, but am instead referring to the peer review system we have in place, where articles only count as “peer reviewed” if they are published by what are regarded as reputable journals.
How did people become so convinced that the current peer review and academic publishing system is so critical and necessary for establishing the quality and credibility of research?
The range of quality for non-reviewed work covers the whole spectrum from raving nonsense to the best published work. But it’s true that there’s no good reason to think the peer review process doesn’t lop off a good chunk of the bottom half of the spectrum, filtering out quite a lot of nonsense; the review process can also tighten up research and arguments, resulting in improvements in what was initially submitted. So the average quality is undoubtedly better than what you’d find outside the process. But does this alone justify the process? No; what you’d need to show is that the process is worth the cost. This would require showing that the advantages outweigh the disadvantages. You’d also want to show that it’s better than alternatives. Neither of these requirements is typically met by anyone defending the present system.
Ironically, there’s little evidence the present system is very good at what it’s supposed to do. And while it would make sense for people to regard it as better than nothing (though it’s not clear to me that it is), in practice many people put far more stock in it than it deserves. They don’t simply use peer review as a heuristic. Instead, they’re often dismissive towards work that isn’t peer reviewed independent of the actual content of the work in question; as if the mere fact that something isn’t peer reviewed is somehow to its discredit. This is stupid and lazy and a sign of bad thinking.
Mandatory pre-publication peer review is relatively recent. Almost all foundational work in science and the humanities predates current institutions. Those institutions weren’t necessary back then, so why treat them as some kind of holy anointing process now? This isn’t, of course, to say that we can’t come to recognize the importance of enshrining a genuinely necessary set of standards that we didn’t once have, but it is entirely legitimate to ask how and why the present system quickly became so entrenched that it’s anathema to question the legitimacy of its centrality.
I also think people don’t realize just how sloppy, haphazard, and lacking in rigor the actual process is. Some editor makes largely private decisions about whether to send a paper out for review or not. Then two or three people who don’t have the status or success to turn down the peer review and are guilted into reviewing it, usually with no compensation at all, are simply told to review it, with almost no guidelines, and certainly no training.
There’s no formal quality control. A review could involve a deep, systematic analysis of the entire article from top to bottom. But almost nobody does that.
It could also involve a graduate student or early career researcher barely competent to assess its contents skimming it or reading it halfheartedly on a random Thursday afternoon in their office and writing four sentences about it. The latter is far more typical. How is that supposed to be the gold standard for credibility? Are you kidding me? Almost nobody checks the data, reruns the code, or deeply evaluates the results or measures. These days, likely a lot of people are just tossing the paper through Claude or ChatGPT to review it, which is probably an improvement over what the typical reviewer achieves.
And ironically, there’s very little peer reviewed research establishing the efficacy of the peer review system we have in place. Almost all of the advocates of peer review don’t know anything about what research has been done anyway, so ironically they’re almost always basing their own support for how central peer review is off of…what? Assumptions? In any case it almost certainly isn’t based on peer reviewed research! So why is it that they feel justified in basing the entire foundation of credibility of all research in academia off of inferences they’ve made in the total or near total absence of supporting peer reviewed data, then leveraging these inferences to insist that peer review is so critical and necessary?
I also want to be clear that there’s a big difference between these claims:
The credibility of research in a given field should be evaluated by competent people and we should have some method or methods for distinguishing work acknowledged as credible from what isn’t that utilize competent/expert evaluation. This is peer review. Peer review is good and we should do that.
We should specifically employ the journal publishing industry to administer peer review and only things that go through this process should be considered “peer reviewed.” Having your work reviewed by any number of experts outside the academic publishing industry simply does not count as “actual” peer review. Even if every single person in your entire field reviewed your work, if it wasn’t done via the formal process of academic journal publishing, it’s just not peer review. Only academic journals are entitled to confer credibility on published work, regardless of the quality of the actual process in any given case.
I fully endorse (1). But (2) is stupid and arbitrary. People frequently conflate the two and don’t understand that I’m opposed to (2) and not (1). The general idea that we should assess the credibility of work and we need capable people to do so is a good one. But this isn’t the actual criterion we use to distinguish work that is “peer reviewed” from work that isn’t. The actual standard is whether work is published in an academic journal. Almost nobody actually seems to be defending peer review itself; what they defend is the practice of publishing in academic journals.
For instance, suppose you took two articles of exactly the same quality, and did the following:
One is submitted to a mid-tier journal with a lackadaisical editor who shrugs and sends it off to reviewers. Most decline. Finally, a graduate student and a middling early career researcher read it halfheartedly. The article cites the researcher, and the graduate student does similar work and finds the content appealing. They both write vaguely positive reviews. The article is published.
One is sent off to five or six excellent and highly competent specialists who work on the topic the article is about. Most ignore it. But one is on sabbatical to write a book on a similar topic and figures “Why not?” They think the article is good, and they give thorough, comprehensive feedback that significantly improves the article. Another reads it carefully and writes a short, approving email stating that they think it’s a solid paper.
By any sane standard, the second article has been more thoroughly “peer reviewed” than the former. And yet only the first article counts as peer reviewed. The second article doesn’t count as peer reviewed at all. “Peer reviewed” doesn’t actually mean “reviewed by one’s peers.” It means it’s been published by a journal after an editor decided it was worth publishing following a review process in which there’s at least some degree of lukewarm approval from one or more academics (who may not even be experts or peers of the author, aren’t required to have graduate degrees, don’t have to have published on the same topic, don’t have their work thoroughly checked, and probably don’t thoroughly check the work they’re reviewing).
Did they read the article? Who knows. Nobody checks. Did they put serious effort into examining it? Probably not, but who knows? Nobody checks. Are they actually experts or peers of the author? Who cares. Horseshoes and hand grenades! As long as they’re kind of in a related field that’s good enough. They aren’t accountable if their reviews are sloppy garbage, and the editor can make virtually any decision they want on the basis of however many reviews of whatever quality they want. The reviews themselves probably won’t be made public nor will the reviewers receive any public credit (or if they do, nobody cares and it’s mostly ignored anyway, so they have no incentive to do good work or care if they do bad work).
There’s little transparency, and it’s very unlikely any aspects of this process will be made publicly available. So if you see the article at all (which isn’t likely since so much is poorly archived and hidden behind paywalls), you won’t know anything about what the reviewers actually thought of the paper. So in addition to being a haphazard, unsystematic, informal process, most of the process is hidden behind emails, editorial decisions that exist exclusively in the minds of editors, and unpublished reviews archived in private accounts that will never see the light of day, ensuring an almost total lack of transparency and a comical lack of consistency, rigor, and quality control.
This is the process people think bestows credibility on publications? This is what many people think distinguishes genuine and credible work from frothing lunacy and tinfoil hats? Are you kidding me?
And yet many people will treat the paper that has gone through the official publication process as a very credible and serious piece of work, while having a highly dubious stance towards the second, i.e., an article that actually received substantive review by peer experts. It’s acceptable to cite the first in scholarly work, but not the second. This is absurd. If you’re a competent scholar and are capable of assessing the merits of an idea, it wouldn’t matter if the idea was scribbled in crayon in a bathroom stall. Yet many people would oppose citing unpublished work even if they knew it was reviewed approvingly by specialists. Indeed, it could be reviewed by all the specialists, and it still wouldn’t matter. After all, it’s not published.
This is idiotic. Heuristics are just that: heuristics. Publication isn’t a magic spell. This kind of lazy, ritualistic way of thinking about credibility is quasi-religious and yet it seems to have entrenched itself in the span of a few decades throughout academia and, by osmosis, here on Substack and the general online debate spaces at the periphery of academia.
And I haven’t even begun to address the fact that bias and other factors may motivate editors to publish some work rather than other work for reasons unrelated to quality, with little oversight, accountability, or means to object. When an article gets desk rejected, your best bet is almost always to move on to the next journal.
Nor have I brought up the fact that journals have an incentive to publish work that will be cited a lot, not merely the best work. Historically, this has promoted publication bias by discouraging failed replications and studies with null results, resulting in a warped and distorted picture of certain bodies of literature that has led to people mistakenly believing in the credibility of entire lines of research that have subsequently been discredited. Note that this has resulted in entire fields believing in fake phenomena for literally decades. This might still happen in the absence of our current institutions, but not only are they not a good filter against this happening, they actively facilitate it.
I also haven’t addressed how journals silo research behind paywalls that make it difficult for people to access and for many impossible to access legally. Even when research is technically accessible, it’s spread in haphazard ways across clunky websites that are hard to navigate, making it difficult to find and access even for academics at times, and even more challenging and in some cases effectively impossible for people without institutional affiliations. This contributes to papers potentially going months or years without anyone reading them. Isn’t the point of publishing work to make it widely available? So why are peer review and publication tied to systems that not only don’t facilitate spreading and sharing information, but actively work to sequester research and hold it for ransom?
Nor have I said that much about how the publication process usually lacks transparency or accountability. Journals are often highly reticent to retract papers, even when they should. It’s also difficult to countermand the decisions of editors…and think about it: even if you’re entirely right, if you make a fuss with an editor, do you think that journal is going to be more receptive to your work in the future, or less? Reviewers suffer little if any punishment for bad work. You can’t be fired for being a bad reviewer, because you weren’t hired in the first place. And consider that: why are the actual experts typically not being paid for their expertise? People go through hell to get PhDs only to receive no compensation for what is effectively expert consultation. A typical lawyer could charge hundreds of dollars an hour to provide fairly standard legal expertise. Yet an expert who may be one of only five or six specialists in the entire world, who received more education, and who is more experienced than a typical lawyer both in general and on the topic in question is given absolutely nothing to labor over a stranger’s work? How does that make any sense? Why aren’t academics charging consulting fees? Why did this become a case in which we’re expected to do an arbitrary amount of free consultation that is, in practice, heaped on those people who can afford the cost of extra labor the least, while the most successful academics and best reviewers are in a better position to turn down these requests? It’s not like we make it a job requirement to do peer reviews and mandate that they be distributed equally. No, the burden is haphazard, uncoordinated, and arbitrarily heaped onto some but not others. That’s the best we can do? In practice, the whole process has some of the trappings of a formal process, but in many ways it is completely informal and arbitrary.
I likewise have said little about how journals often take months or even years to publish research. This stifles progress, but it also throws up so much red tape that good work may go unpublished. It also rewards people who have the particular personality traits needed to push their work through the process, while discouraging those who don’t, which isn’t a quality filter so much as a personality filter. At the same time, it incentivizes “publishable” research. As a result, journals serve to establish certain kinds of work as credible or mainstream and certain other kinds of work as unconventional. Since career advancement, status, prestige, and grant funding all center on publishing in journals, this creates strong incentives to work on topics that journals are more receptive to. This can stifle innovation and undermine creativity. There are studies showing many researchers explicitly and very consciously state that they don’t work on the topics they prefer or at least do less work on those topics precisely for this reason.
And I haven’t added how the incentive systems this puts in place more generally lead to the extreme amplification of Goodhart’s law: that any measure that becomes a target ceases to be a good measure. The whole purpose of the peer review process is to serve as a quality filter. And yet when you turn the whole process into a competition where everyone involved’s ambitions hinge on publication, you create an incentive to publish for the sake of publishing itself. Researchers would be better off publishing three or four middling or even bad articles in journals than they’d be doing one slower, more careful project of higher quality. They’d be better off pursuing a weak project that they know journals in their field would be eager to publish than a strong project that would be hard to publish.
At the same time, there are powerful incentives due to Goodhart’s law to jockey to get your name on papers as a coauthor even if you’ve contributed little or nothing. This leads to a distorted and dishonest literature where names are attached to papers that don’t belong there. Bowing to such pressure can corrupt the people who do so. If you’re dishonest about this, why not be dishonest in other ways, too? After all, nobody is harmed by unjustified authorships, are they? This is a rhetorical question. Yes, everyone is harmed by it.
There are incentives to use AI to hasten your publication rate, even if you churn out slop. I saw but have yet to investigate a report that nearly a third of recent dissertations in STEM fields included AI-generated text. There’s incentive to cut corners, do easier research with a faster turnover rate, fail to take time to polish work because of deadlines (a certain number of publications may be required or expected to keep or obtain a job) or internal pressure (more publications are always better than fewer, all else being equal).
Given all these considerations I haven’t even begun to address in any detail, it may very well be that the norm of mandatory pre-publication peer review and the peer review process itself not only haven’t improved the quality of research, but have actively led to undermining the quality of research. But the rot doesn’t stop there. They’ve centered the entirety of academia around the process. As a result, they threaten not only the quality of research, but the quality of researchers themselves. We’re dragged along, kicking and screaming, by the yawning maw of the journal publication system, because our advancement and even continued participation in our respective fields require that we pass through this process.
We’ve created a system that isn’t actually measuring quality. It’s measuring an indirect indicator of quality. And because of the conflations mentioned at the start of this article, people are so lazy, stupid, naive, or negligent that they’re simply unwilling to look at the situation for what it is: we’ve let our metrics run amok, and have replaced an actual and direct evaluation of the merits and quality of work with a tool initially devised to serve as a public indicator of quality.
This process may have worked well initially, but with the gradual entropy associated with any system vulnerable to being Goodharted, exacerbated by the highly competitive job markets throughout nearly every academic field, the resulting system is one that not only no longer does a good job of serving its initial purpose, but one that has warped and corrupted academia and academics themselves.
I see little way out. The system has become so entrenched that not only has the measure become the target (i.e., the goal is publication, not the production of good research), it’s become so integral to career advancement that the costs of anyone abandoning ship are too massive, and individuals and smaller groups couldn’t defect without being marooned somewhere outside academia, where few would ever hear from them again. You can either participate in the process and be consumed by it, or you can be left behind.
It’s as if we all started out racing towards a bright finish line, only for it to gradually dawn on us that we were barreling towards a monster. And yet, despite the fact that many of us can see the many-tentacled horror looming on the horizon, we have to keep racing towards it. After all, we wouldn’t want our colleagues to get eaten first.