Test coverage is largely used to assess test effectiveness. In practice, not all code is equally important for coverage analysis, for instance, code that will not be executed during tests is irrelevant and can actually harm the analysis. Some coverage tools provide support for code exclusion from coverage reports, however, we are not yet aware of what code tends to be excluded nor the reasons behind it. This can support the creation of more accurate coverage reports and reveal novel and harmful usage cases. In this paper, we provide the first empirical study to understand code exclusion practices in test coverage. We mine 55 Python projects and assess commit messages and code comments to detect rationales for exclusions. We find that (1) over 1/3 of the projects perform deliberate coverage exclusion; (2) 75% of the code are already created using the exclusion feature, while 25% add it over time; (3) developers exclude non-runnable, debug-only, and defensive code, but also platform-specific and conditional importing; and (4) most code is excluded because it is already untested, low-level, or complex. Finally, we discuss implications to improve coverage analysis and shed light on the existence of biased coverage reports.

I am a Professor in the Department of Computer Science at UFMG, Brazil. Research interests: software evolution, software repository mining, software quality, and empirical software engineering. I received my PhD in Computer Science from the University of Lille / Inria, France. During three years, I was a Professor at FACOM/UFMS. Before, I was a Postdoctoral researcher at the ASERG/UFMG group. I was also a software developer at Inria, Lille, France and a research intern at Siemens, Erlangen, Germany.