Major Revisions at ESEC/FSE 2023
By Kelly Blincoe and Paolo Tonella, ESEC/FSE 2023 Program Chairs, and Satish Chandra, ESEC/FSE 2023 General Chair
SE conferences are currently forced to reject papers that include valuable material, but would need major changes to become acceptable for conference publication, because major revisions cannot be accommodated in the current review process. By supporting only a binary outcome, conferences force reviewers to decide between rejection and acceptance even in borderline cases that would be better judged after a round of major revision. Some of these decisions could have just as well gone the other way. The rejected borderline papers that include valuable ideas and material are usually improved and submitted to the next conference. When there is value in the submission, this process converges and eventually the paper gets accepted and presented at a conference. But this comes with several drawbacks: (1) the evaluation process is restarted each time, ignoring previous comments by reviewers and the way they have been addressed by the authors; (2) the community dedicates unnecessary effort for the reviewing of these borderline papers, which might eventually get assigned 6 or 9 reviewers instead of the 3 that would be necessary to accompany the paper to acceptance; (3) the authors get conflicting and inconsistent reviews at each new re-submission, making the convergence process troublesome and uncertain.
ESEC/FSE 2023 will introduce Major Revisions to address all three above mentioned issues, at least in cases where the problems affecting the submission can be fixed in a single iteration of major revisions. We hope that by allowing major revisions the review process will be overall more predictable and reliable and that we can accept more papers.
Review Process
When converging to their final decision in the discussion phase, reviewers will be offered three possible outcomes: accept, reject, major revision. Papers that would have been accepted at conferences that do not allow major revisions should still be accepted — major revisions should not lower the initial acceptance rate. Papers that include potentially publishable material but require major changes (e.g., additional experiments or new analyses of existing results; major rewriting of algorithms and explanations; clarifications, better scoping and improved motivations) are eligible for the major revision outcome. Authors of these papers are granted 8 weeks to implement the requested changes. In addition to the revised paper, authors are asked to submit a response letter that explains how each required change was implemented. If any change was not implemented, authors can explain why. The same reviewers will then review the revised paper and make their final (binary) decision.
As there is no quota for the accepted papers, there is also no quota for major revisions. However, we expect that thanks to major revisions ESEC/FSE 2023 will be able to eventually accept 10–15% more papers, while keeping the quality bar absolutely unchanged.
With the possibility of a major revision outcome, meta-reviews become extremely important. The meta-review should clearly and explicitly list all major changes required by the reviewers to make the paper acceptable for publication. The meta-review should act as a contract between reviewers and authors, such that when all required changes are properly made, the paper is accepted. In this respect, the listed changes should be extremely clear, precise, and implementable.
Major revision is not shepherding. While shepherding typically focuses on important but minor changes, which can be specified in an operational way and can be checked quite easily and quickly by reviewers, major revisions require major changes (although doable in 8 weeks), which means the instructions for the authors cannot be completely operational and the check will need to go deeply into the new content delivered by the paper. Hence, while the expectation for shepherded papers is that most of them will be accepted once the requested changes are implemented, this is not necessarily the case with major revisions. Prior editions of ESEC/FSE used conditional accept for minor revisions. This year, we will encourage the PC members to accept such cases. In the case that the PC members feel the required changes must be rechecked, these papers will go through the major revision process.
Criteria for Major Revisions
Major revision should not become the default choice for borderline papers and should be used only in cases that meet the following criteria:
● Without major revisions, the paper would be rejected. However, the concerns raised by the reviewers are such that a properly done major revision, which addresses the reviewers’ concerns, would make the paper acceptable for publication;
● The requested changes should be doable in 8 weeks and should be implementable within the page limit;
● The requested changes should be strictly necessary for paper acceptance (i.e., not just nice-to-have features);
● The requested changes require recheck (i.e., reviewers cannot trust the authors to implement them directly in the camera ready).
Examples of good reasons for a major revision include:
● Comparison with some recent work or an important baseline was omitted and the reviewers agree that it should be possible to carry out such comparison in 8 weeks time;
● The presentation requires substantial rework and improvement, e.g. by describing key algorithms/concepts better, by reorganising existing material, or by clarifying the motivations;
● Results are weak, but in 8 weeks additional experiments can be run to strengthen them;
● Methodology has minor problems that can be fixed in 8 weeks;
● More discussion on SE problems/needs is needed and can be reasonably added in 8 weeks;
On the other hand, there are cases when rejection is a more appropriate outcome than major revision. Here are some examples:
● The requested additions/changes are not implementable in 8 weeks;
● The contribution is very narrow or not relevant to the SE audience, and it cannot be retargeted in 8 weeks;
● The methodology is flawed and cannot be fixed in 8 weeks;
● Results are unconvincing; the paper does not seem to improve the state of the art much, and new convincing results are unlikely to be available after 8 weeks of further experiments;
● The customary benchmark used in the community was ignored and cannot be adopted and compared to in 8 weeks.
Sometimes a straight accept decision is more appropriate than major revision. We do not want major revision to become the primary pathway for acceptance. We should continue to trust the authors to make minor changes to the submissions in the camera ready version. For instance, acceptance is preferable when:
● The requested additions/changes are nice to have features, not mandatory for the acceptability of the work;
● Minor improvements of the text are needed;
● Minor clarifications requested by the reviewers should be incorporated;
● Important but not critical references should be added and discussed;
● Discussion of results could be improved, but the current one is already sufficient.
Final remarks
Changes in customary processes are difficult to achieve. On the other hand, we believe the SE community should continually reflect on and improve its practices. By experimenting with major revisions in the ESEC/FSE 2023 review process we hope to contribute to such self-reflection and continuous improvement exercise, which can succeed only if the entire community is listened to, before, during and after implementing the change. We know we will make mistakes, but we are confident that as a community we will be able to fix them and we will eventually deliver a better conference review process. So, feedback is welcome!
Disclaimer: The posts in the SIGSOFT Blog are written by individual contributors and any views or opinions represented in their posts are personal, belong solely to the blog authors and do not necessarily represent those of ACM SIGSOFT or ACM.
Call for Contributions: Please consider contributing to the SIGSOFT Blog.
