Why adopt a double-blind peer-reviewing process?
Several studies have shown the existence of (unconscious) biases in the peer reviewing process in computer science. Both positive and negative biases have been identified. For instance, in a controlled experiment at WSDM 2017 (50% of the PC single-blind, the other 50% double-blind), Tomkins et al. (2017) found that
- single-blind reviewers enter fewer bids;
- single-blind reviewers preferentially bid on papers from top universities and companies; and
- single-blind reviewers are more likely to assign a positive score to a paper of a famous author or from a top university or company.
WSDM is a computer science venue; in the absence of a controlled study in the real-time community, it is reasonable to assume that similar unconscious biases exist in our community.
Other studies have further shown biasing effects related to author gender, author ethnicity, author nationality etc. Lee et al. (2013) provide a comprehensive review of the scientific literature on bias in the peer reviewing process.
Recently, Le Goues et al. (2018) discussed the effects of introducing a double-blind peer-reviewing process at ASE, OOPSLA, and PLDI, and arrived at two important conclusions: first,
“while anonymization is imperfect, it is fairly effective,”
“the extra administrative burden [is] relatively minor and well worth the benefits.”
For these reasons, the IEEE TCRTS has decided to introduce a double-blind peer-reviewing process at RTSS 2019.
Is the real-time community large enough for an effective double-blind peer-reviewing process?
How are Conflicts of Interests (CoIs) discovered?
- Authors must indicate any TPC members with whom they have CoIs as well as the types of CoIs.
- Authors also need to notify the PC Chair of exceptional CoIs up to the submission deadline.
- The submitter is responsible to collect the CoIs of every author of the submission and then enter them into the system before the submission deadline.
- Authors may optionally indicate PC members that they have a COI with that is outside of the normal scope of COIs. This information, including proper reasons for the additional COIs must be communicated directly to the Program Chair in writing prior to the submission deadline. These PC members will not normally be eligible to review the authors’ paper(s). Note the final decision on whether to make this restriction rests with the Program Chair.
- In addition to the regular bidding phase, PC members will be asked to declare CoIs based on a list of authors and affiliations. This list will not include paper titles, abstracts, or paper IDs.
How will reviewers judge whether a contribution is “incremental”?
One frequent concern is that reviewers may have a hard time judging whether a paper’s contribution is substantial enough to merit publication at a top conference if they do not know the authors’ prior work. In the process adopted for RTSS 2019, this is not actually a problem, for two reasons:
- Author identity does not determine whether a contribution is incremental; rather, the delta relative to the state of the art does. A contribution does not become less (or more) incremental only because it extends one’s own prior work.
- Since references are not blinded, authors can (and must) discuss the state of the art objectively, and reviewers will be able to fully assess the state of the art before arriving at their decision.
How will reviewers judge whether a contribution is correct?
In some discussions, the point has been raised that it can be difficult to place much trust in the correctness of a proof without knowing the identity of the authors. This is exactly the kind of positive or negative identity-based bias that a double-blind reviewing process seeks to eliminate. Either a proof is clear and convincing, or it is not. “Proof by authority “ is not an acceptable scientific method, nor are prior mistakes sufficient justification for rejection of new results.
Similarly, some have worried that it may be difficult to weigh the validity of experiments without knowing the authors or lab behind a study. Again, a study’s merits, limitations, and weaknesses should be clearly apparent from (and only from) the submitted paper. Author identity plays no role in the assessment of empirical work.
What if some papers can be de-blinded anyway?
No double-blind peer-reviewing process is perfect. Papers by the best-known authors with distinctive styles can still remain recognizable to many. Similarly, authors working for a long time on highly specialized topics that few other authors work on will be recognizable to area experts. There are many other ways in which submissions might occasionally become de-blinded (by a subset of the reviewers), e.g., in case of a resubmission of a paper previously rejected from a single-blind conference or due to a pre-existing online preprint and an honest search for related work by a reviewer working on a similar topic. Such cases cannot always be prevented.
The important point, however, is that, to be beneficial overall, a double-blind peer-reviewing process does not have to be perfect. In particular:
- A double-blind process allows authors who believe that they might be at a disadvantage to craft a paper that cannot be attributed to anyone in particular with high likelihood.
- The fact that the process is double-blind changes reviewers’ attitudes and (subconsciously) reminds everyone that (suspected) author identity should play no role in the assessment.
- Double-blind submissions encourage authors to present the state of the art and prior work in a balanced way.
- Even if a paper is likely authored by a certain author or group, the fact that it plausibly could also have been authored by someone else in most cases creates enough uncertainty to deemphasize the role of author identities in the decision-making process.
Can a paper on arXiv be still submitted to RTSS following the double-blind policy?
Yes, it is ok to submit a paper on arXiv to RTSS. The submission should still follow the double-blind policy. However, during the submission, the author must send a note as well as the arXiv version to the PC chair to indicate that the paper has been put on arXiv.
The PC chair will instruct TPC members not to deliberately search for any papers unless they suspect there might be duplicated submissions. If they happen to see the same paper on Arxiv, they should inform the PC chair but should not reveal the identify of the authors to the other TPC members.
The bottomline is that if one puts her/his paper on arXiv, s/he does bear the risk of being known to be the author of a submission. However, the person will not be considered as violating the double-blind requirement.
What is the double-blind policy for reviewers?
Reviewers will not intentionally try to de-blind a paper (e.g., by searching for key phrases on the web). Nonetheless, as discussed above, there may still be situations where some of the reviewers feel like they have a pretty good idea about who (some of) the authors of a paper assigned to them may be. In such situations, the following rules apply.
- If PC members feel that they can de-blind a submission based on title and abstract alone (e.g., because they reviewed a prior version of the paper for a single-blind conference, or because they had a hallway conversation with the authors at a some conference prior to submission, etc.), they may still bid for the paper and should not declare a conflict of interest (at least not for this reason) because their expertise is still required.
- If PC members feel that they can de-blind a submission for any reason, they should not reveal their suspicion to anyone and should not let it influence their assessment of the work.
- In particular, if a PC member happens to have reviewed a prior version of a submission for another conference, the PC member should not reveal this information and should be careful to base her/his assessment exclusively on the present version.
What about track chairs?
The double-blind policy does not extend to Track Chairs (one of whom is the Program Chair). Since track chairs organize the peer-reviewing process, they know all author identities and affiliations (of papers submitted to their track), know all reviewers, assign papers to reviewers, and lead the final discussions.
Give that the Track Chairs do not themselves review papers but rather facilitate discussion, authors are not required to indicate CoIs with the track chairs. However, if a Track Chair indeed has CoIs with some authors, the respective papers will be handled by the other Track Chair or a senior, non-conflicted member in the same track.