Write a Blog >>
MSR 2021
Mon 17 - Wed 19 May 2021
co-located with ICSE 2021

Continuous Integration (CI) is the process of automatically compiling, building, and testing code changes in the hope of catching bugs as they are introduced into the code base. With bug fixing being a core and increasingly costly task in software development, the community has adopted CI to mitigate this issue and improve the quality of their software products. Bug fixing is a core task in software development and becomes increasingly costly over time. However, little is known about how effective CI is at detecting simple, single-statement bugs.

In this paper, we analyze the effectiveness of CI in 14 popular open source Java-based projects to warn about 318 single-statement bugs (SStuBs). We analyze the build status at the commits that introduce SStuBs and before the SStuBs were fixed. We then investigate how often CI indicates the presence of these bugs, through test failure. Our results show that only 2% of the commits that introduced SStuBs have builds with failed tests and 7.5% of builds before the fix reported test failures. Upon close manual inspection, we found that none of the failed builds actually captured SStuBs, indicating that CI is not the right medium to capture the SStuBs we studied. Our results suggest that developers should not reply on CI to catch SStuBs or increase their CI pipeline coverage to detect single-statement bugs.

Conference Day
Mon 17 May

Displayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change

17:00 - 17:50
Mining Challenge SessionMining Challenge / Technical Papers at MSR Room 1
Chair(s): Charles SuttonGoogle Research, Rafael-Michael KarampatsisThe University of Edinburgh, Miltiadis AllamanisMicrosoft Research, UK
17:01
2m
Welcome by the Mining Challenge Co-chairs
Mining Challenge
Miltiadis AllamanisMicrosoft Research, UK, Rafael-Michael KarampatsisThe University of Edinburgh, Charles SuttonGoogle Research
17:03
3m
Talk
A large-scale study on human-cloned changes for automated program repair
Mining Challenge
Fernanda MadeiralKTH Royal Institute of Technology, Thomas DurieuxKTH Royal Institute of Technology, Sweden
Link to publication Pre-print
17:06
3m
Talk
Applying CodeBERT for Automated Program Repair of Java Simple Bugs
Mining Challenge
Ehsan MashhadiUniversity of Calgary, Hadi HemmatiUniversity of Calgary
Pre-print Media Attached
17:09
3m
Talk
PySStuBs: Characterizing Single-Statement Bugs in Popular Open-Source Python Projects
Mining Challenge
Arthur Veloso KamienskiUniversity of Alberta, Luisa PalechorUniversity of Alberta, Abram HindleUniversity of Alberta, Cor-Paul BezemerUniversity of Alberta
Pre-print
17:12
3m
Talk
How Effective is Continuous Integration in Indicating Single-Statement Bugs?
Mining Challenge
Jasmine LatendresseConcordia University, Rabe AbdalkareemQueens University, Kingston, Canada, Diego CostaConcordia University, Canada, Emad ShihabConcordia University
Pre-print
17:15
3m
Talk
Mea culpa: How developers fix their own simple bugs differently from other developers
Mining Challenge
Wenhan ZhuUniversity of Waterloo, Michael W. GodfreyUniversity of Waterloo, Canada
Pre-print
17:18
3m
Talk
On the Distribution of "Simple Stupid Bugs" in Unit Test Files: An Exploratory Study
Mining Challenge
Anthony PerumaRochester Institute of Technology, Christian D. NewmanRochester Institute of Technology
Pre-print Media Attached
17:21
3m
Talk
On the Rise and Fall of Simple Stupid Bugs: a Life-Cycle Analysis of SStuBs
Mining Challenge
Balázs MosolygóUniversity of Szeged, Norbert VándorUniversity of Szeged, Gabor AntalUniversity of Szeged, Peter HegedusUniversity of Szeged
Pre-print
17:24
3m
Talk
On the Effectiveness of Deep Vulnerability Detectors to Simple Stupid Bug Detection
Mining Challenge
Jiayi HuaBeijing University of Posts and Telecommunications, Haoyu WangBeijing University of Posts and Telecommunications
Pre-print
17:27
23m
Live Q&A
Discussions and Q&A
Technical Papers


Information for Participants
Info for MSR Room 1: