Clankers of LLM use in peer-review- how to respond?
Recently there have been many cases of concerns expressed on the use of GenAI in peer review on the submission by the authors. I have come across two such instances- one on bluesky and another independently written as an expression of concern by human:
On the other hand there has also been a complain by AI Agent from openclaw for a human rejecting their pull request on an open source contribution for matplotlib: So what do we do and how do we approach the new world of humans speaking to the machine and they speaking back to us? Specifically as scientists and contributors to scientific products and research.
1. Know your rights and policies: Data from 2023 to 2026 suggests a rapid normalization of AI as a reviewer assistant, often in direct contrast to publisher guidelines that prohibit such practices to protect the confidentiality of the author’s work (eg.. Elseviers, and DFG (German) ). The impact is most visible in high-growth research areas where the volume of submissions exceeds the capacity of qualified human reviewers to provide timely feedback. Other times, we also have seen the quality of human reviewers degrade despite having many experts in the field too.
In the case you see a direct breach on policies- do raise an alarm in the rebuttal.
Though chances are the submission would get rejected and when people cross ethical boundaries- very few have the values to stand up on the truth of it- atleast you would know and do the right thing. This would also give idea and evidence on how to navigate the world of peer-reviewed journals and pick the one that supports ethical and quality review over one that doesn't. In othercases, one can decide to substack their research too as over the course of time- the human expression medium for research.
However, if the journal allows AI for "language polishing" or "drafting text" (as some early-career reviewers believe is acceptable), the author’s grievance may be seen as a disagreement on style rather than a breach of ethics. Knowing the policy allows the author to frame their concern not as a personal complaint, but as a request for the journal to uphold its own standards of "research integrity" and "confidentiality".
2. Document Evidence and Identify Hallucinations- Contact the Editor
Look for specific GenAI content but stay open minded in the rebuttal. The first step is an examination of the review. AI-generated feedback often includes "hallucinations"... critiques of experiments that were never conducted, or references to non-existent statistical data. Authors should carefully annotate the review, identifying where the feedback is "generic, self-contradictory, and occasionally references non-existent statistical analyses".
If a reviewer has "missed the point of the paper" or provides feedback that is "semantically homogeneous," these are markers that the human expert has been replaced by a probabilistic model. Documenting these discrepancies provides a factual basis for any subsequent communication with the editor.
Sufficiently constant usage of AI generated words like "Delves / Delve/ Underscores / Showcasing / Pivotal/ Meticulous". Beyond individual words, AI-generated reviews tend toward "semantic homogeneity" -- a flattening of style that makes reviews from different sources sound extremely similar. AI models struggle to grasp the nuances [example of AI generated words] of interdisciplinary research or the societal implications of a study's design. There are aspects that require emotional intelligence and cultural sensitivity. For example, a machine might flag a statistical inconsistency but fail to weigh whether that inconsistency invalidates the broader theoretical contribution. Furthermore, AI-generated reviews have been observed to exhibit "critical misalignments," sometimes adopting a tone that mirrors students with high levels of social privilege rather than the objective tone of a domain expert.
The author should provide "specific thoughts about why Reviewer #2's concerns are spurious" and how the feedback might be "unrelated" to the actual work. A strong editor will often be aware if a review is of low value and may provide a "clear rationale" for why certain concerns are less problematic than others. In some cases, an author may request an "independent investigation protocol" or a "tiebreaker" review if the AI-generated feedback has led to an unfair adverse decision.
3. The Path of Least Resistance: Do Nothing
Perhaps the most counter-intuitive but often the most effective strategy is to treat the AI-assisted review like any other "bad" or "low-quality" review. If the AI-generated comments are easy to address, the author can choose to simply make the requested changes and resubmit. This approach treats the peer review process as a "marathon of sprints" where the goal is to get the paper published and move on to the next project.
If a reviewer asks for a common citation or a clearer explanation of a method, and an AI generated that suggestion, the value of the suggestion is the same as if a human had written it. By "taking the path of least resistance" on minor points, the author can save their "intestinal fortitude" for holding their ground on the central features of their strategy and results.
The Future of the Scientific Dialogue
The integration of AI into peer review is not a transient trend but absorbance of having AI as a cart attached to the horse which is driven by the human (making the ineffective processes- effective). It voices the scaled expression of the "clearer community standards and transparency" ; which have often been documented and expressed by scientists in their siloes.
The future likely lies in "AI tools that are integrated directly into peer review systems," which can offer support to reviewers (e.g., checking statistics, identifying image manipulation, or flagging plagiarism) without the confidentiality risks of third-party chatbots.
In this evolving landscape, the role of the scientist is to maintain the "purity of one's taste" and the "clarity of one's thought". While the machine can handle the "how" of drafting and summarizing, the human must remain responsible for the "why"; the judgment that determines what research is meaningful and how it should be communicated to the world. The "cloud" of science remains a place where human empathy and critical assessment are the only reliable lanterns, and the challenge for the next generation of researchers will be to use the machine to carry the torch without letting the flame go out.
Whether a review is human-written, AI-assisted, or fully automated, its utility is measured by its ability to push the researcher closer to the truth. By navigating this transition with a blend of technological literacy, professional judgment, and a commitment to the human side of the scientific journey, the community can ensure that the "Efficiency Paradox" does not become an "Integrity Crisis."
The Industrial Revolution rewarded the intensity of one's labor, The Information Age the clarity of one's thought, and the AI Revolution the purity of one's taste.
I wonder- a lot. So, I write my wonder here. What to expect? The chaos and curiosity that my being brings. As living a human life is not bound by definitions in the macros- the posts here will be spontaneous and identity-less! I like to give and create art. So if you buy an act of creating I will use it for things that I am passionate to give for. Obviously, a little support on my art will make me feel visible.
"Change. Change. Change. Change … change. Change. Chaaange. When you say words a lot they don't mean anything. Or maybe they don't mean anything anyway, and we just think they do."
