On the Use of Mutation in Injecting Test Order-Dependency
Background: Test flakiness is identified as a major issue that compromises the regression testing process of complex software systems. Flaky tests manifest non-deterministic behaviour, send confusing signals to developers, and break their trust in test suites. Both industrial reports and research studies highlighted the negative impact of flakiness on software quality and developers’ productivity. While researchers strive to devise solutions that could help developers addressing test flakiness, the elaboration and assessment of these solutions are hindered by the lack of datasets large enough to leverage learning techniques. Aim: To address this lack, we conduct an exploratory study that investigates a new mean for producing datasets of flaky tests. Method: We propose an approach that relies on program mutation to inject flakiness in software tests. In particular, we plan to delete helper statements from tests to make their outcomes order-dependent, i.e., pass in certain running orders but fail in other orders. We intend to apply our mutation-based approach to a set of 14 Java projects to assess the effectiveness of test mutation in injecting order-dependency and generate a new dataset that could be used to study flakiness.
Mon 17 MayDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
10:00 - 10:50 | Testing and code reviewTechnical Papers / Data Showcase / Registered Reports at MSR Room 2 Chair(s): Jürgen Cito TU Wien and Facebook | ||
10:01 3mTalk | A Traceability Dataset for Open Source Systems Data Showcase Mouna Hammoudi JOHANNES KEPLER UNIVERSITY LINZ, Christoph Mayr-Dorn Johannes Kepler University, Linz, Atif Mashkoor Johannes Kepler University Linz, Alexander Egyed Johannes Kepler University Media Attached | ||
10:04 4mTalk | How Java Programmers Test Exceptional Behavior Technical Papers Diego Marcilio USI Università della Svizzera italiana, Carlo A. Furia Università della Svizzera italiana (USI) Pre-print | ||
10:08 4mTalk | An Exploratory Study of Log Placement Recommendation in an Enterprise System Technical Papers Jeanderson Cândido Delft University of Technology, Jan Haesen Adyen N.V., Maurício Aniche Delft University of Technology, Arie van Deursen Delft University of Technology, Netherlands Pre-print Media Attached | ||
10:12 3mTalk | Does Code Review Promote Conformance? A Study of OpenStack Patches Technical Papers Panyawut Sri-iesaranusorn Nara Institute of Science and Technology, Raula Gaikovina Kula NAIST, Takashi Ishio Nara Institute of Science and Technology Pre-print | ||
10:15 4mTalk | A Replication Study on the Usability of Code Vocabulary in Predicting Flaky Tests Technical Papers Guillaume Haben University of Luxembourg, Sarra Habchi University of Luxembourg, Luxembourg, Mike Papadakis University of Luxembourg, Luxembourg, Maxime Cordy University of Luxembourg, Luxembourg, Yves Le Traon University of Luxembourg, Luxembourg Pre-print Media Attached | ||
10:19 3mTalk | On the Use of Mutation in Injecting Test Order-Dependency Registered Reports Sarra Habchi University of Luxembourg, Luxembourg, Maxime Cordy University of Luxembourg, Luxembourg, Mike Papadakis University of Luxembourg, Luxembourg, Yves Le Traon University of Luxembourg, Luxembourg Pre-print Media Attached | ||
10:22 28mLive Q&A | Discussions and Q&A Technical Papers |
Go directly to this room on Clowdr