![[PukiWiki]](image/wadanyan.png "[PukiWiki]")
ISSRE2010
21st IEEE International Symposium on Software Reliability Engineering
http://www.issre2010.org/
http://www.issre2010.org/content/student-papers
Submission deadline Extended!!!!!!!!
タイトル：
原稿：
An_Analysis_of_Committers.pdf
カメラレディ原稿: [添付]
スライド： [添付]
accept!!!
Originality of the Work: 3
Overall Quality: 2
Presentation and English: 3
Overall recommendation: 2
This paper proposes an analysis of a large open source project, namely postgresql, to quantify the quality of the contributors, and better understand their activity. The authors divide contributors into committers and non-committers, and quantify their skills by looking at their responsiveness in the exchanges in the mailing lists.
I like the idea underlying this paper. The objective of the authors is certainly ambitious and difficult to achieve. However, the solution proposed in the paper can be considered only a small first step: it is a big simplification, so big as to impact the credibility of its conclusions.
First and foremost, the authors start from the assumption that timeliness in generating patches is always a synonymous of quality. This is not true in my opinion. Some bugs may be very easy to fix: unexperienced contributors may thus focus on these problems and quickly produce patches to them. Other bugs may be much more complex to fix, and may for instance require modifications to the application core. Skilled developers may choose to focus only on these problems, and leave the surface ones to the less skilled contributors. According to the assumptions introduced in this paper, these unexperienced contributors would be of the highest quality! If reaction speed is to be taken into account for the evaluation of the quality of the work of an individual, the easiness of the modification must be taken into account too. A small GUI change cannot be compared to a radical change to the application core.
Also, the authors never take into consideration the contributor timezone. In large projects such as postgresql the contributors are likely to be located all over the world. Their location highly impacts their responsiveness: a 12 hours delay may be solely due to the fact that the patch is reported by a Chinese developer, and edited by an american one.
Finally, committers are likely to be the most stable contributors, because of a personal hobby or passion. It is therefore absolutely normal that, over the whole mailing list lifetime, they are also the most prolific. It would be much more interesting to normalize the number of produced patches with respect to the period of activity.
In short, while the underlying idea is absolutely valid and may lead to interesting results, the authors have oversimplified a bit too much the problem. I would strongly invite the authors to continue this work and add the required sophistication to make the ultimate results more meaningful.
Originality of the Work: 3
Overall Quality: 3
Presentation and English: 2
Overall recommendation: 4
The paper studies the patch review process of the PostgreSQL database in order to characterize the “performance” of committers and non-committers in terms of the time they take to develop a patch and the number of patches they developed.
The study presented in the paper is very interesting and is backed by real data, which makes it even stronger. However, I found the paper presentation somewhat lacking and have some criticism regarding the assumptions of the study.
In terms of presentation, the use of English is not good. I would suggest the authors to find a native speaker or someone with more experience in order to correct some “style bugs” on the paper (e.g., use Figure 1 instead of Figure1) . Moreover, since figure 1 presents a cyclic process (a patch can be edited and reviewed many times before being committed), it would be better to represent it as a flowchart or using other more formal method to represent loops.
One of the key assumption of the work when comparing developers is that all patches require the same effort. Besides the fact that committers can be more responsible, thus they spend more time making their patches (as pointed in Section VI), for me, it is clear that committers solve the most critical problems, the ones that affect the core of the system. This work would be greatly improved if the authors try to classify the applied patches in terms of complexity (even if the classification is not perfect). One way to do that is to look at patch description or the discussions regarding the bug(s) it is solving. Another way to do that is to interview some of the developers in order to identify the most complex parts of the code and use this information to classify these patches. The important point here is that patch complexity are a fundamental information to classify developers working on a OSS project based on the number of patches they are adding.
Originality of the Work: 3
Overall Quality: 3
Presentation and English: 3
Overall recommendation: 3
The paper studies the patch review process of the open-source PosgreSQL database system.
I found the main aims of the study very interesting and it is clear that a substantial amount of effort has been spent in processing and analysing the data.
However, the analyses could be improved by analysing in more detail some of the confounding variables that may effect the analysis. You could look into ways of measuring the "size" of the bug and resulting patch (by looking at the number of lines of code that are changed in the patch or the number of files that need to be changed as part of the patch). This may then allow you to derive better measures of the good reviewers and "patchers" rather then just looking at their response times. Also check how sensitive the results are to the different time zones that the reviwers are working at. Do you get a completely different ordering if you factor in the timezones?
Overall, despite the short commings, I believe the paper would be fine as a Student paper, and would benefit from the discussion in the conference. Though the above shortcommings would have to be addressed before submiting the paper to a regular paper stream.
