Thinkers debate whether AI is likely to produce groundbreaking insights in philosophy or in other, less important fields such as mathematics and the sciences. But it’s clear that what LLMs are good at right now and will consistently be better at than groundbreaking insights are helping to produce barely-publishable work, academic slop.
Reconfiguring our understanding of reality or changing the trajectory of an entire field is hard. Writing an article tweaking an existing problem slightly or finding one more counterexample in an ongoing debate is far, far easier. And it’s something LLMs’ recombination abilities are well set up to do. This worry isn’t merely abstract. Just this week, one major journal in a different field, Organization Science, released its findings on AI use in submissions. The journal’s AI Task Force found that submissions are up over 40% post-ChatGPT and that over 100% of this increase comes from AI-generated content. In recent months over 40% of the journal’s total submissions were significantly or entirely generated by AI (submissions that score 30-100% on AI-assistance).1 While AI-detection is imperfect, the trends here are so clear and so strong over a wide number of submissions that they’re hard to doubt.
Significantly AI-generated manuscripts (score of over 30% ) were 60-80% more likely to be desk rejected and high-AI submissions (over 70%) were 3 times less likely to receive the most common positive outcome, Revise & Resubmit. Although the rejection rates were higher, these papers were not uniformly desk-rejected. 3.2% of high-AI papers still received Revise & Resubmit decisions (that’d be a pretty decent success rate for many philosophy journals). Whether your chances of publication go up or down if you use AI becomes a matter of running the numbers. If, using AI, you can write more than 3 times as many papers as you did unaided, your overall chances of publication probably go up, even if your average acceptance percentage goes down! The incentive to try AI is even higher if your baseline acceptance rate is lower than average: if you’re not succeeding on the old pre-AI system, you’ve got little to lose.
Turning academic journals into venues for AI-generated articles is one huge issue. But even if all AI-generated submissions were rejected by the peer review process, they’d still have done significant harm by taking up more and more of the available attention of peer reviewers. The peer review system was already in danger of failing and hard to justify. If reviewers now spend half their efforts on AI-generated manuscripts that system becomes completely unsustainable. And the more often reviewers are given largely AI-generated manuscripts, the more reviewers might see AI-generated reviews as permissible or at least highly tempting.
Given this situation, it’s time to return to Jennifer Whiting’s 2015 proposal for slow philosophy:
Tenure and promotion evaluations of candidates’ research will be based exclusively on what the candidates select as their best work.
Candidates pick the 50 to 80, 000 words (3-5 articles or a book) of their publications they deem to be of the highest quality. Additional publications are not considered at all by the relevant review committees. This removes the incentive to publish more and more. Now the incentive is to wait, reflect, and only publish when your idea is as well-formulated as you can make it. Going forward, candidates are only incentivized to publish a new article if it’s better than their current five best articles. The proposal would reset incentives from quantity to quality.2
This proposal has other advantages too. Jennifer Whiting originally put it forward so that candidates would have an “easier time balancing work with other responsibilities (including civic and family ones).” Those concerns remain just as relevant. And, as I noted, the advantages of this proposal for reviewers are more and more relevant. At a time when it’s harder and harder to find referees, the request would become easier when they are only asked to look at the best a candidate can do, not every possible item of academic value.
The main worry from the commentators on Whiting’s original post was that it penalizes academics who do lots of good-enough work. Here I agree with Whiting that the current system is so strongly tilted towards this sort of productivity that bending the stick the other way is well worth doing. There might also be certain types of academic work, such as public philosophy, where quantity is what departments want to incentivize. Those could simply be evaluated separately as part of a candidate’s overall portfolio.
What reasonable limitations would look like will vary a lot with discipline. But in pretty much every discipline it’s much easier for AI-assistance to increase production of barely publishable work than increase ground-breaking insights. So there’s a strong case for almost every academic discipline to consider developing a version of slow academia. In the age of AI, we need to increase the care we pay to the most important work we’re doing. Peer review should focus attention on our best work instead of incentivizing human scholars to produce more and more to keep up with AI-generated scholarship.
The authors’ model was set up to give an “AI assistance score” from 0 to 1, based on analyzing segments of the text separately, with scores between 30% and 70% crossing “the threshold from primary human ownership to giving the AI substantially more control” and over 70% counting “as primarily AI-generated content.” While these estimates have some degree of error or uncertainty, the authors report that altering categorizations did not significantly change the overall results. They also did not find evidence of Pangram classifying text written by non-native English speakers as AI-generated (texts from such authors submitted in the pre-ChatGPT were not flagged as AI-generated).
For those always suspicious of the author’s motives, my pre-tenure and promotion record of publication quantity was perfectly fine for someone teaching a 4-4 load. I’m already a full professor at my institution, which, as a teaching focused school, has a clear and relatively low bar for scholarly activity for tenure and promotion.