A Framework for Evaluating the Results of the SZZ Approach for Identifying Bug-Introducing Changes

Authors - Daniel Alencar da Costa, Shane McIntosh, Weiyi Shang, Uirá Kulesza, Roberta Coelho, Ahmed E. Hassan
Venue - IEEE Transactions on Software Engineering, Vol. 43, No. 7, pp. 641–657, 2017

Related Tags - TSE 2017 software quality defect prediction

Abstract - The approach proposed by Śliwerski, Zimmermann, and Zeller (SZZ) for identifying bug-introducing changes is at the foundation of several research areas within the software engineering discipline. Despite the foundational role of SZZ, little effort has been made to evaluate its results. Such an evaluation is a challenging task because the ground truth is not readily available. By acknowledging such challenges, we propose a framework to evaluate the results of alternative SZZ implementations. The framework evaluates the following criteria: (1) the earliest bug appearance, (2) the future impact of changes, and (3) the realism of bug introduction. We use the proposed framework to evaluate five SZZ implementations using data from ten open source projects. We find that previously proposed improvements to SZZ tend to inflate the number of incorrectly identified bug-introducing changes. We also find that a single bug-introducing change may be blamed for introducing hundreds of future bugs. Furthermore, we find that SZZ implementations report that at least 46% of the bugs are caused by bug-introducing changes that are years apart from one another. Such results suggest that current SZZ implementations still lack mechanisms to accurately identify bug-introducing changes. Our proposed framework provides a systematic mean for evaluating the data that is generated by a given SZZ implementation.

Preprint - PDF


  Author = {Daniel Alencar da Costa and Shane McIntosh and Weiyi Shang and Uirá Kulesza and Roberta Coelho and Ahmed E. Hassan},
  Title = {{A Framework for Evaluating the Results of the SZZ Approach for Identifying Bug-Introducing Changes}},
  Year = {2017},
  Journal = {IEEE Transactions on Software Engineering},
  Volume = {43},
  Number = {7},
  Pages = {641–657}