Internal validity threats concern factors that may affect a dependent variable and were not considered in the study. We paid attention not to violate the assumptions of the statistical tests that are performed in the paper. Specifically, in RQ1 and RQ2, we applied non-parametric tests that do not require making assumptions on the distribution of our dataset.
Threats to internal validity have to do with whether other plausible hypotheses could explain our results. We assume that fluctuations in the performance and impactful properties of JIT models are linked with fluctuations in the nature of fix-inducing changes. However, other intangible confounding factors could be at play (e.g., changes to team culture, contributor turnover). On the other hand, we con- trol for six families of code change properties that cover a broad range of change characteristics.
Our findings may be specific to nonlinear logistic regression models, and may not apply to other classification techniques. As suggested by our prior work [8], we are actively investigating other classification techniques like Random Forest. Preliminary results indicate that the findings of RQ1 can be reproduced in the Random Forest context. However, since the Wald χ2 importance scores are not well-defined for such classifiers, a common importance score needs to be identified before RQ2 and RQ3 can be replicated.
External validity threats are concerned with the generalizability of our results. In this paper, we only studied Mozilla Firefox. First, Mozilla Firefox is the most studied system for issues related to rapid releases; moreover, the system’s data are publicly available. We also have the opportunity to per- form both quantitative and qualitative analyses (including the interviews with release managers) on this system. However, we should recognize that our findings may not be generalizable to other systems. In the future, we plan to collaborate with other software organizations, to validate and extend the results of this work. In addition, more studies on other systems with other programming languages are suitable to further validate our results. To facilitate future replication studies, we share our datasets and scripts at: https://github.com/swatlab/uplift- analysis.
Threats to external validity have to do with the generalizability of our results to other systems. We focus our study on two open source systems. We chose strict eligibility criteria, which limits the systems that are available for analysis (see Section 3.1). Due to our small sample size, our results may not generalize to all software systems. However, the goal of this paper is not to build a grand theory that applies to all systems, but rather to show that there are some systems for which properties of fix-inducing changes are fluctuating. Our results suggest that these fluctuations can have a large impact on the performance of JIT models and the quality improvement plans that are derived from JIT models. Nonetheless, additional replication studies are needed to generalize our results.
Construct validity threats are concerned with the relation- ship between theory and observation. Previous studies [19], [8] suggested that complex code is a good indicator of fault- proneness. We confirm this point in this study. However, we found that fault-inducing patches are more likely to be submitted by experienced developers, which contradicted our expectations. We attribute this outcome to the fact that ex- perienced developers are often assigned to difficult issues, whose resolution tend to be more complex. Also, release managers might overlook risks associated to patches submitted by experienced developers, as these developers are often more trusted than others.
![[PukiWiki]](image/wadanyan.png "[PukiWiki]")
