What’s New with ICSE 2022

By Daniela Damian and Andreas Zeller, ICSE 2022 Program Chairs, and Matthew Dwyer, ICSE 2022 General Chair

What was new in the ICSE 2022 review process? What is new in how the actual conference is run this year? Below we offer a glimpse into our review process and on how we are planning the virtual and in-person ICSE 2022!

We set up the reviewing process to cope with the high submission numbers and reviewing load — by having a large number of dedicated program committee members, but also by setting up special measures to ensure that reviews would be of high quality and that all papers would be held to the same standard. Overall, this resulted in another ICSE record of 197 papers accepted — an acceptance rate of 28.5%.

Explicit and Transparent Reviewer Guidelines

Building on the example from Amy Ko and Jan Vahrenhold at ICER 2021, ICSE 2022 defined a set of reviewer guidelines that were iterated over in collaboration with the PC, and shared with the authors prior to submission deadline. These guidelines 1) described the technicalities of the reviewing task and discussed the criteria by which papers should be reviewed — soundness, significance, novelty, verifiability and transparency, and presentation, and 2) included patterns to follow and anti-patterns to avoid. These guidelines helped tremendously in getting high-quality reviews, notably by asking reviewers explicitly to cover all criteria in their reviews; they also allowed common ground among the reviewers to provide consistent paper evaluation. Our reviewer guidelines have been universally lauded in social media — not only in Software Engineering, but also in neighboring fields. We hope that they continue to be helpful to program committees in future years and to provide the foundation of a transparent, consistent, collaborative and collegial review process at ICSE.

Autonomy to Area Chairs

We further refined the roles of ICSE Area Chairs for the conference. Our seven area chairs Lionel Briand, Tevfik Bultan, Massimiliano Di Penta, Sonia Haiduc, Anita Sarma, Frank Tip, and Andrea Zisman all effectively acted as chairs for the 100+ papers assigned to each — reading reviews and guiding reviewers towards a consensus, such that they could make the final decision on each paper. In contrast we, the actual PC Chairs, did not have to decide on any paper (although we would have had the last word); instead our focus was on general guidelines and reviewer assignments. ICSE 2022 may be the first Software Engineering conference in which all involved in the paper final decision (reviewers and Area Chairs) were ‘blind’ to the paper authorship, and where those who knew the papers authors (the PC Chairs) would not decide on a paper. This enhanced autonomy placed on ICSE Area Chairs resulted in significant workload and we are very grateful for their dedicated, consistent response to what their role demanded. We could not have had a better Area Chairs team this year!

Special Emphasis on Open Science Policies

As in previous years, we supported the adoption of open data and open source principles and encouraged all contributing authors to disclose (anonymized and curated) data to increase reproducibility and replicability. Although the sharing of research data was not mandatory for submission or acceptance, sharing was expected to be the default, and the authors were expected to justify non-sharing. We recognized that qualitative research or in industrial studies might not be governed by goals of reproducibility or replicability and that they often face challenges in sharing research data. To help reviewers assess the reliability and credibility of research that used qualitative methods or was conducted in industrial studies, we provided additional, explicit author recommendations for the reporting of research methodologies, procedures and analyses.

With Gratitude, to Our Program Committee

What few people realize until they have served on the ICSE PC is the amount of time and dedication that goes into reviewing but also paper discussions. This year we continued best practices of previous ICSEs, such as assigning at least three reviewers to each paper, double-anonymous reviewing, asking authors to respond to reviews, and having reviewers produce meta-reviews that would summarize the decision. We took extra care that all reviews and meta-reviews be checked for quality, and that authors’ comments were explicitly taken into account. Our PC members were highly engaged, and discussed the tiniest technical details — our longest discussion consisted of no less than 93 messages. Awards are given out to reviewers who were especially dedicated.

We would like to take this opportunity to offer our thanks to all members of the ICSE PC for their enthusiasm, dedication, and service to the ICSE community. They wrote over 2,785 reviews, 691 metareviews, placed over 11,000 comments, made over 2,200 edits to improve their reviews — all while also teaching, advising their students, and taking care of their families during the COVID-19 pandemic. They conducted this work voluntarily and invested many hours of their time reading and discussing papers, and writing reviews.

A Three-phase, Online and In-person ICSE 2022 Program

In preparing the program, we are accounting for the challenges and possibilities of the brave new world we live in by splitting the conference into three phases.

• In Phase 1, all papers of all tracks are available to all attendees, accompanied by 20-minute videos each (aiming for availability by end of April). This allows attendees to select and study the material at their leisure (the “Netflix” model) in advance of the conference; it also allows attendees to promote their work online.
• Phase 2 is the virtual conference (May 10–13) where the focus is not to present full talks, but rather to ignite discussions. Distributed across three time bands, a blended program will consist of 5-min paper presentations in 6-paper sessions. The goal is to accommodate papers with short talks and ample space for questions and interaction. To foster interchange further, each paper will be presented in two different time bands in sessions with different paper authors. The Keynote talks will be featured in the virtual program.
• Finally, there will be an In-Person conference (May 22–27) where the focus is on meeting and interacting. A tentative In-Person Program Schedule is already online. Attending authors are invited to present their papers live, again as short 5-min presentations followed by ample discussion, as well as in the form of posters. We also plan to organize Birds of a Feather (BoF) sessions that discuss questions prepared by keynote speakers or around topics proposed by the community. There are no capacity limits to BoFs, and the venue has many large rooms. Participants will be asked to indicate an interest in BoF sessions, but are not obliged to participate. Community meetings, e.g. SIGSOFT/TCSE Town halls will take place at the end of the day, and other meetings, e.g. for affinity groups, will be over lunch. In-person Workshops and Co-located events will precede ICSE, Sunday through Tuesday, while some events will only have virtual programs.

These three phases are larger than ever: The final schedule for the virtual conference blends no less than 800 presentations! While preparing the schedule, we needed to ensure that authors present in two time bands that are convenient for them, are not assigned to two sessions running at the same time, and are grouped by topic. These constraints make manual scheduling a Herculean task. Our newly appointed scheduling chair, Johannes Lampel, has written a program that takes all the talk and authors’ data and produces an almost optimal solution within a few minutes.

A Final Word

Adapting and innovating on our ICSE 2022 process and program would not have been possible without those before us: we thank the ICSE 2021 Program Co-Chairs Arie van Deursen and Tao Xie for their valuable help and insights for us to carry out the program co-chair duties. We also thank all members of the ICSE Steering Committee for their support, and the track chairs for their incredible work.

Our most important thanks, however, go to all of the 2500+ authors who submitted their papers to the ICSE 2022 Technical Track! We look forward to meeting and facilitating great discussions in May, and hope that everyone will draw lots of insight and inspiration from our program!

Disclaimer: The posts in the SIGSOFT Blog are written by individual contributors and any views or opinions represented in their posts are personal, belong solely to the blog authors and do not necessarily represent those of ACM SIGSOFT or ACM.