[[ふじこ]]
* IWESEP2010 [#r79a5577]
#contents
IWESEP2010~
International Workshop on Empirical Software Engineering in Practice~
[[http://iwesep.wordpress.com/:http://iwesep.wordpress.com/]]~
~
- Paper Submission: October 11, 2010
- Acceptance Notification: November 8, 2010
- Camera Ready Copy: November 22, 2010
~
タイトル：An Analysis of Committers Activities in the Patch Review Process of OSS Development~
原稿：&attachref(IWESEP_paper_提出版.pdf);~
** 予定 [#b4efb562]
- 原稿
-- 10/11原稿提出
* 査読結果 [#n87e29b1]
rejected
---------------------------- REVIEW 1 --------------------------
- Reviewer 1
In this paper, the authors look at the process by which patches are editted and
reviewed before being accepted into an OSS project or rejected. They measure
the rate that different groups edit and review patches and the time taken to
review and edit patches. Unfortunately, there are a number of major flaws with
the paper that make it unsuitable for publication in it's current form. I have no
doubt, however, that if changes are made (and I'm including my suggestions here),
a revision of this paper would be of publication quality.
My biggest problem with this work is that is omits two important prior studies
on the subject. Rigby et al published a very in depth study on the patch
review process in Apache at ICSE 2008. Bird et al published a study that
examined what factors influence someone becoming a core developer (two of the
factors examined were patch submission and acceptance), which examined a number
of OSS projects and include PostgreSQL. The fact that there is great overlap
in the goal of these studies and that they are neither discussed nor do the
authors indicate what differentiates their study from prior studies is a large
point against this paper.
The authors make a number of claims that are either incorrect or
unsubstantiated and do not provide references. For instance, they claim that
increasing committers speeds up development because they can bypass review and
commit directly to the project repository. In a number of OSS projects, this
is incorrect. Most OSS projects, still review and discuss almost all changes
even if they are made by core developers. Rigby's paper has an in-depth
discussion of this phenomenon. The authors also claim that normally, a
developer is selected as a committer through nomination as a result of
admirable contributions. This is correct, but a reference should be cited.
One prime example of a good supporting reference would be Nicholas Duchenaut's
ethnograhic study of the python project. The authors say that it has not been
easy to elect an appropriate deveoper as a committer from a number of
developers. Is this true? What evidence is this based on? If this is an
informed guess, then the author's should state that. The fifth paragraph of
section 6 includes a claim that NPR, NPR, PRT, and PET have to be evaluated to
select a committer depending on the state of the project and/or product
properties. Why? The author's have shown that there is a difference between
the two, but they have not examined anything about how committers are selected.
In the sixth paragraph, the authors claim that the restricted number of
committers is unique to PostgreSQL as opposed to other projects. I've seen
many projects that limit committers. There is no reference given or even
examples of projects that don't restrict committers.
I *LOVED* figure 1. This is a wonderful depiction of the patch submission and
review process and the terms used in the paper. Good job. The only thing I'd
add is the acronyms PRT and PET since they are depicted and used in the paper,
but do not appear on the figure.
The data collection is short on details. How were the patches collected? In
my experience, I've found that sometimes patches are attached to messages and
sometimes inlined in the text of a message. Did the authors get both types of
patches? Did they use heuristics? How was this done. The authors also only
selected developers who had reviewed or edited patches at least 5 times. I can
imagine why, but it would be useful to *explain* to the reader why this was
done and why the number 5 was chosen as the threshold. In section 5.2.3, the
committers are classified using "criteria". As a reader, I am left only to
speculate what that criteria is. This needs to be explained. The term
criteria is mysteriously also used in the third paragraph of section 6. What
does it mean in this context?
I applaud the use of the Mann-Whitney test for examining the data. A t-test
would be inappropriate because it is clearly right skewed. However, the
author's should both explain why they used that test and also give a reference
so that those readers less familiar with non-parametric statistics can
understand what is going on.
I found the results in section 5 to be interesting, but their presentation
suffered some problems. The four paragraphs in section 5.2.1 are simply copies
of each other with names and numbers changed. That's not useful, and almost
disrespectful to the reader. I rarely harp on the writing itself unless it is
unclear, but I feel that in this case, the authors should be more concise and
possibly discuss the implications of each result. A simple table could present
the data in those paragraphs more succinctly in be more easy to reference.
Lastly, the authors claim that the analysis in this paper was performed
to improve the patch review process in OSS. They haven't shown how
their results improve this process. From the results, we know some more
about the relationship of timing in subsmission and review with roles
within the project. I would stay away from making bold claims without
the evidence to back them up.
The major flaws of unsubstantiated claims, important missed related work,
missing important details, and the minor flaw of some presentation issues
need to be addressed before this paper is accepted for publication.
---------------------------- REVIEW 2 --------------------------
- Reviewer 2
The paper presents a study of the patch review process in PostgreSQL. More specifically, it compares the characteristics of committers and non-committers with respect to time it take to review patches, revise patches, and number of reviews per month. The findings are interesting and the novelty is fine for a workshop. The paper could be improved by better relating to other work on patch review (for example by Peter Rigby and colleagues, ICSE 2008, Technical Reports, etc.). To some extent the findings also reflect that most OSS has a small core team, which drives the development and takes contributions from outside.
Abstract: Write "number of patch reviews" instead of "NPR"
Introduction: The authors write "In general, the modifications to fix existing bugs are carried out as follows". However, there many unique ways to fix bugs and even in open-source I would argue that there is no general way.
Section 2: Do all emails about patches get responses? (This is also relevant for the analysis in Section 4.)
Section 5.2.3, last paragraph: The paper writes by observing Figure 6 that "Therefore almost [all] committers tend to review and edit patches more than non-committers". Maybe I misunderstood something, but isn't this in contrast to Figure 4, which shows more reviews/edits for non-committers?
---------------------------- REVIEW 3 --------------------------
- Reviewer 3
This paper studies the differences between committer and non-committer in
open-source software development. It provides a case study of ProgreSQL for
finding the differences based on times and frequencies for reviewing and editing
patches.
Overall, I liked the idea and it has the potential to be of a great value for
OSS developers. However, the setup of the case study weakens the work presented
in this paper.
I have some points:
The authors state one of the motivations for clarifying the differences between
committers and non-committers is for supporting the nomination of new committers.
However, as also mentioned by the authors, the role of committers and
non-committers is different; the differences may be resulted from the roles
themselves. This fact becomes stronger for projects having superreviewers (the
2nd-round reviewers). To be more precise this, I think the data of
non-committers should be compared with ones of committers _before_ they were
promoted to committer. I want it will conclude the same; the work of committers
is also great when they didn't have the commit permission. Nevertheless,
clarifications are important.
The explanation of the patch review process in Section 2 lacks the generality of
the process. Of course, it fits some OSS projects including PostgreSQL. But I
can easily imagine some possible variability such as: superreviews, testing or
reproductions of bugs by developers/users, rejections, and of course, deep
discussions. The author should discuss which tasks are important for their aim
and the build the process model.
Minor points:
--- The definitions of committer and non-committer are redundant.
* 備考・その他 [#pc140a03]
#comment
![[PukiWiki]](image/wadanyan.png "[PukiWiki]")
