http://ipkitten.blogspot.com/2020/09/can-academic-peer-review-learn.html
All patent attorneys used to be scientists. Many of us hold a PhD. Working closely with scientists may lead us to wonder what life would have been like had we chosen to stay in science. However, for this Kat at least, nostalgia for the lab is quickly quashed by any exposure to the peer-review process. Patent prosecution, with its to and fro of structured, fair and reasoned argument can be an immensely satisfying process. Peer-review, by contrast, seems almost purposely designed to frustrate and dampen the spirit of scientific endeavour. However, both peer-review and patent prosecution have similar ends, namely determining whether a patent or scientific manuscript makes a meaningful contribution to the art. Perhaps then, peer-review could learn something from patent prosecution?
Patent prosecution and peer review
Both peer-review and the patent prosecution, of course, have their problems. Peer-review is considered, to paraphrase, the worst system for reviewing academic papers except for all the others. However, the comparison between peer-review and patent prosecution puts many of the problems of peer-review into sharp relief, and perhaps sheds some light on why science struggles to retain those from unrepresented groups.
Patent prosecution is a codified system in which professional patent office examiners determine whether a patent should be granted. The examiner assesses the claims of the patent application for novelty, inventiveness, and utility, as defined by the patent law. If the examiner raises an objection, the applicant is given some number of months to respond to the objection, either by arguing their case or by amending the claims. The to and fro between the applicant and the examiner may go on for many iterations, before final acceptance or rejection.
Peer-review is similar to patent prosecution except that the place of the examiner is taken by experts in the field, who are presumed to be in the best position to assess the truth, originality and significance of the research. Manuscripts submitted to a journal by an author are sent by the journal editor to expert academics in the field to review. Like the patent examiner, the peer-reviewer passes judgement on the merits of the paper and recommends publication, potentially subject to revisions, or rejection. The author is then invited by the editor to respond to the reviewer’s comments and to make the suggested revisions. The editor takes the final decision on whether the paper should be accepted or rejected based on the reviewer comments and the author’s reply.
Before beginning the comparison between peer-review and patent prosecution, it is noted that academic purists may be scandalised by any attempt to equate the world of peer-review with the commercial world of patents. However, such critics may be reminded that published papers are the currency of science, on the basis of which grants are awarded, jobs offered, and promotions given. A published paper therefore has economic value to its authors, albeit a more indirect one than a patent. The process by which papers are accepted and rejected should thus be as arguably fair and robust as those by which a patent is granted.
Why anonymity in peer-review matters
| Science can be stressful |
In patent prosecution, the identity of the applicant is known, as is the identity of the professional examiner. By contrast, peer-review relies on anonymity, or a least a pretence of anonymity. Ideally the reviewer should thus not be influenced by either their prior knowledge of the author, or by the fact that the author will know their identity.
However, in practice, true anonymity in peer-review is rarely achieved. Many authors self-publish their papers on arXiv before submitting to a peer-review journal. Reviewers can thereby easily check the author’s identity. Even without online pre-prints, it is usually clear to anyone who knows the field from whom a paper originates, given that an author will often build upon their own previous work. Reviewers will also be keen to bring the author’s attention to the reviewer’s own work for citation, but by doing so, will unmask their identity.
So why does it matter if the identity of the author is clear to the reviewer and vice versa? Peer-reviewers, like patent examiners, are supposed to provide objective comments on the scientific merits of the work. However, a reviewer’s objective judgment may be clouded if they suspect the identity of the author. Reviewers are likely to look more favourably on papers originating from eminent figures in the field. Reviewers will also be unlikely to advise rejection of papers originating from high profile companies or university departments at which they may be interested in future employment. On the flip side, reviewers may be prejudiced against certain authors with whom they have professional (or heaven-forbid, personal) disagreements. The peer-review system therefore provides the potential for personal and/or professional biases to cloud a reviewer’s assessment of a paper.
By contrast, in the patent system, the identities of the examiner and applicant are known to each other. Anonymity is not required because examiners are not potential competitors of the applicants, and do not work in the field.
Peer-review operates on vague rules
The problems that may occur because of the lack of anonymity in the peer-review process are compounded by the lack of clear, codified criteria for assessing a paper for scientific merit. This lack of clarity contrasts with the patent system, in which novelty and inventiveness is assessed according to objective tests.
Peer-review should assess a paper in a similar way to how a patent examiner assesses the claims of a patent application, i.e. does the research relate to something that is new, non-obvious and useful? Depending on the journal, peer-review should also assess the potential impact of the research, i.e. is it something that a lot of people in the field would be interested in. However, in contrast to patent prosecution, the assessment criteria for much of peer-review is left vague. There is no definition of what it means for research to be non-obvious or impactful. This is left to the discretion of each and every reviewer.
Worse still, peer-reviewers are not, unlike patent examiners, trained review experts. Peer-reviewers are encouraged by editors to produce long reviews. The result is that peer-reviewers will produce reams of nitpicking comments (for example, on writing style, figure format and terminology), whilst not addressing the critical question of whether the paper is making a important contribution to the art. The peer-review system is thus one in which reviewers may be influenced by personal and professional prejudices that are not curtailed by a codified system of assessment to which the reviewer must abide. Again, this contrasts with the patent system, in which the rules and criteria are well-established, relatively consistent and known to both the applicant and the examiner.
Peer-review provides only a limited opportunity to reply
Like the patent system, the peer-review system permits authors to respond to a reviewer’s objections. The time given for response varies from months to a few days. There is often a word limit for response (e.g. limited to only a page) and authors may not be permitted to submit a corrected manuscript. Furthermore, reviewers will often ignore the author’s response, in which case the judgment of the original review stands. By contrast, patent prosecution not only permits but legally requires that an applicant be given the opportunity to respond to an examiner’s objections (EPO Guidelines for Examination, B. Right to be Heard).
The patent system of professional examiners and codified rules thus removes a lot of the problems associated with peer-review. Patent objections are clearly defined by the law, as is the method and processes for redress. Compared to the peer-review system, patent applicants have relative clarity as to the criteria the application must satisfy, and the tests that should be used for assessment. Having professional examiners also largely removes ego and potential conflicts of interest from the equation.
The ugly side of peer-review
To be a scientist, you need a thick skin. Rejection and failure are part of the daily grind of science, whether it is the failure of experiments or the numerous rejections of manuscripts and grant proposals that must be endured before (hopefully) final success. The uncodified process of peer-review exacerbates this problem. The propensity of reviewers to be both excessively critical of inconsequential details of the paper whilst simultaneously vaguely casting doubt on the authors general capabilities as a scientist, can seriously dampen a scientist’s self-esteem. It is therefore of little surprise that unrepresented groups, who may lack self-confidence, are more likely to leave science than those from other groups. The patent system has its own problems, but at least is a far more impersonal system than peer-review.
Perhaps then, peer-review could learn something from patent prosecution. The patent system and peer-review have similar aims. The purpose of patent prosecution is to establish whether an author’s work should be awarded the credibility of publication in the journal, effectively a green tick from the journal that work is new, non-obvious and useful. The purpose of patent prosecution is to establish whether an applicant should be awarded an IP right for a new, non-obvious and useful invention. The patent system demonstrates that it is possible to objectively test for concepts such as inventiveness and utility. Why then could the peer-review system not use similar tests? A more formalised template for analysis in peer-review would help to remove ego from the equation and minimise the time spent on extraneous matters.
Reading some example peer-reviews (courtesy of Mr Kat) renders this Kat eternally grateful for the patent profession. Thankfully, she can leave behind the peer-reviewers and return to her task of explaining politely but firmly why an examiner should grant a patent according to the established principles of patent law. If the academic peer-review process could only adopt a few of the practises of patent prosecution it could perhaps become a similarly fair and rational process.
Content reproduced from The IPKat as permitted under the Creative Commons Licence (UK).