![[PukiWiki]](image/wadanyan.png "[PukiWiki]")
[[ふじこ]] ***質疑メモ [#d27c5b6e] - Peter Popov [Session Chair](City Univ) -- "it is too normal that committers have higher NPR and NPE than non-committers"は本当？ --- committersのほうが参加期間が長い場合が多いので、だいたいの場合は本当 -- RQ3の分類にnon-committersを含めてやると面白いかも --Q: "it is too normal that committers have higher NPR and NPE than non-committers"は本当？ ---A: committersのほうが参加期間が長い場合が多いので、だいたいの場合は本当 --comment: RQ3の分類にnon-committersを含めてやると面白いかも - Loizos Markides (Seattle Univ) -- medianよりも多い（時間と回数）開発者が大量にいた場合は？ --- medianだけでcommitters選出を行うには現状では厳しいから、future workなどでいくつか詳細にみるべき -- average NPR and NPE per "month" →月平均は妥当？ --- とりあえず今回のpostgreSQLでは・・・。 --Q: medianよりも多い（時間と回数）開発者が大量にいた場合は？ ---A: medianだけでcommitters選出を行うには現状では厳しいから、future workなどでいくつか詳細にみるべき --Q: average NPR and NPE per "month" →月平均は妥当？ ---A: とりあえず今回のpostgreSQLでは・・・。 ***渡航の詳細など… [#o6f2cdce] - フライト~ OSAKA/KANSAI 31 OCT 10 SUN 1735 UNITED UA 886 SAN FRANCISCO 1108 FLIGHT TIME- 9:33 SAN FRANCISCO 05 NOV 10 FRI 1140 UNITED UA 885 OSAKA/KANSAI 06 NOV 1550 FLIGHT TIME-12:10 - ホテル~ Ramada Limited San Jose Convention Center ~ Address: 455 South 2nd Street San Jose CA United States ~ Phone: 1-408-2983500 ~ Fax: 1-408-2982477 ~ URL: http://www.ramada.com ~ ~ - ISSRE開催場所~ Cisco Campus~ Address: 225 East Tasman Drive, San Jose, California 95134~ Campus Map: http://www.issre2010.org/sites/default/files/cisco_san_jose_campusmap_v1.pdf * ISSRE2010 [#r79a5577] #contents ISSRE2010~ 21st IEEE International Symposium on Software Reliability Engineering~ [[http://www.issre2010.org/:http://www.issre2010.org/]]~ [[http://www.issre2010.org/content/student-papers:http://www.issre2010.org/content/student-papers]]~ Submission deadline Extended!!!!!!!!~ ~ - Student Paper Submission: %%July 15%% August 15, 2010 - Acceptance Notification: %%August%% September 2010 - Camera Ready Copy: %%August 30%% %%September 24%% October 1, 2010 ~ タイトル：An Analysis of Committers toward Improving the Patch Review Process in OSS development~ 原稿：&attachref(An_Analysis_of_Committers.pdf);~ カメラレディ原稿: &attachref(An_Analysis_of_Committers_CameraReady.pdf);~ 班内発練スライド：&attachref(ISSRE606_v05.pptx);~ 全体発練スライド：&attachref(ISSRE606_v08.pptx);~ 発表スライド：&attachref(Shoji_ISSRE606_v10.pptx); ** 予定 [#b4efb562] - 原稿 -- 7/30日本語版原稿（IPSJ関西支部大会）完成 -- 8/4英語第１版 -- 8/11英語第２版 -- 8/15完成 -- 10/1カメラレディ原稿提出 - 発表準備 -- 10/12-15 班内スライドチェック -- 10/20 班内発練 -- 10/22 バーカー先生チェック -- 10/27 全体発練 - 発表 -- 10/31 渡米 -- 11/1-4 当日 -- 11/5 帰国 -- 11/6 日本着 * 査読結果 [#n87e29b1] accept!!! --------------------------------------------------------------------------- ** Reviewer #1's Scores [#gc465023] --------------------------------------------------------------------------- Originality of the Work: 3 Overall Quality: 2 Presentation and English: 3 Overall recommendation: 2 --------------------------------------------------------------------------- This paper proposes an analysis of a large open source project, namely postgresql, to quantify the quality of the contributors, and better understand their activity. The authors divide contributors into committers and non-committers, and quantify their skills by looking at their responsiveness in the exchanges in the mailing lists. I like the idea underlying this paper. The objective of the authors is certainly ambitious and difficult to achieve. However, the solution proposed in the paper can be considered only a small first step: it is a big simplification, so big as to impact the credibility of its conclusions. First and foremost, the authors start from the assumption that timeliness in generating patches is always a synonymous of quality. This is not true in my opinion. Some bugs may be very easy to fix: unexperienced contributors may thus focus on these problems and quickly produce patches to them. Other bugs may be much more complex to fix, and may for instance require modifications to the application core. Skilled developers may choose to focus only on these problems, and leave the surface ones to the less skilled contributors. According to the assumptions introduced in this paper, these unexperienced contributors would be of the highest quality! If reaction speed is to be taken into account for the evaluation of the quality of the work of an individual, the easiness of the modification must be taken into account too. A small GUI change cannot be compared to a radical change to the application core. Also, the authors never take into consideration the contributor timezone. In large projects such as postgresql the contributors are likely to be located all over the world. Their location highly impacts their responsiveness: a 12 hours delay may be solely due to the fact that the patch is reported by a Chinese developer, and edited by an american one. Finally, committers are likely to be the most stable contributors, because of a personal hobby or passion. It is therefore absolutely normal that, over the whole mailing list lifetime, they are also the most prolific. It would be much more interesting to normalize the number of produced patches with respect to the period of activity. In short, while the underlying idea is absolutely valid and may lead to interesting results, the authors have oversimplified a bit too much the problem. I would strongly invite the authors to continue this work and add the required sophistication to make the ultimate results more meaningful. - 論文のベースとなっているアイデアは面白い - 目的は野心的で，達成するのが難しいが，この論文のsolutionは小さな第一歩だと考えられる -- ただし，結果の信憑性に影響を与えるくらい大幅に簡略化しすぎている - パッチの作成にかかる時間を品質と（ほぼ）同義に扱っているが，そうではないと思う -- 初心者のcontributorsは，修正が簡単なバグに取り組んで短時間でパッチを生産するかもしれないし -- 逆にコアなcontributorsは複雑なバグにのみ集中し，簡単なバグを初心者に任せるかもしれない -- 反応速度で品質を測るのならば，修正の容易さも同時に考慮しなければいけない - 時差による要因を考慮していない -- 特に12時間以内の遅れは開発者のエリアによるものかもしれなく，PostgreSQLのような世界中に開発者の存在するプロジェクトではなおさら - コミッターは（非コミッターに比べて）長期間に渡って活動している可能性が高く，回数が多くなるのは当然のように思える -- 一定期間あたりの回数を数えたほうが面白い - この研究の根底にある考えは有効で面白い結果に繋がると思われるが，この論文では問題をあまりに単純化しすぎた --------------------------------------------------------------------------- ** Reviewer #2's Scores [#j442bdca] --------------------------------------------------------------------------- Originality of the Work: 3 Overall Quality: 3 Presentation and English: 2 Overall recommendation: 4 --------------------------------------------------------------------------- The paper studies the patch review process of the PostgreSQL database in order to characterize the “performance” of committers and non-committers in terms of the time they take to develop a patch and the number of patches they developed. The study presented in the paper is very interesting and is backed by real data, which makes it even stronger. However, I found the paper presentation somewhat lacking and have some criticism regarding the assumptions of the study. In terms of presentation, the use of English is not good. I would suggest the authors to find a native speaker or someone with more experience in order to correct some “style bugs” on the paper (e.g., use Figure 1 instead of Figure1) . Moreover, since figure 1 presents a cyclic process (a patch can be edited and reviewed many times before being committed), it would be better to represent it as a flowchart or using other more formal method to represent loops. One of the key assumption of the work when comparing developers is that all patches require the same effort. Besides the fact that committers can be more responsible, thus they spend more time making their patches (as pointed in Section VI), for me, it is clear that committers solve the most critical problems, the ones that affect the core of the system. This work would be greatly improved if the authors try to classify the applied patches in terms of complexity (even if the classification is not perfect). One way to do that is to look at patch description or the discussions regarding the bug(s) it is solving. Another way to do that is to interview some of the developers in order to identify the most complex parts of the code and use this information to classify these patches. The important point here is that patch complexity are a fundamental information to classify developers working on a OSS project based on the number of patches they are adding. - 研究内容は非常に面白く，実データによって検証されているので良い - 論文の書き方にいくつか問題がある -- 英語がよくない．ネイティブか英語の熟練者を捕まえてチェックしてもらえ -- 図１のパッチレビュープロセスの図はフローチャート（もしくは他の何か）のほうがよい --- レビューとパッチ編集は場合によってはループするものだから，フローチャートで描くほうが正確に描ける - この論文では全てのパッチは同じもの（評価基準）として扱われてる -- コミッターは責任があることなどから，より重大なバグに着手することは明らかなので，パッチの複雑さで分類を行うべき --- バグに対する説明や議論を見てみる --- バグの修正にどれだけの人が関わったか（議論も含めて） --- などで複雑さを分類できる --------------------------------------------------------------------------- ** Reviewer #3's Scores [#wd850085] --------------------------------------------------------------------------- Originality of the Work: 3 Overall Quality: 3 Presentation and English: 3 Overall recommendation: 3 --------------------------------------------------------------------------- The paper studies the patch review process of the open-source PosgreSQL database system. I found the main aims of the study very interesting and it is clear that a substantial amount of effort has been spent in processing and analysing the data. However, the analyses could be improved by analysing in more detail some of the confounding variables that may effect the analysis. You could look into ways of measuring the "size" of the bug and resulting patch (by looking at the number of lines of code that are changed in the patch or the number of files that need to be changed as part of the patch). This may then allow you to derive better measures of the good reviewers and "patchers" rather then just looking at their response times. Also check how sensitive the results are to the different time zones that the reviwers are working at. Do you get a completely different ordering if you factor in the timezones? Overall, despite the short commings, I believe the paper would be fine as a Student paper, and would benefit from the discussion in the conference. Though the above shortcommings would have to be addressed before submiting the paper to a regular paper stream. - 研究の狙いが面白い -- データ分析に相当な量のeffortがなされていたこともわかります - より詳細に分析を行うべき -- パッチの規模（パッチの行数や影響しているファイルの数など） -- パッチの規模と反応時間を見ると関係がわかるかも - レビュアーのタイムゾーンによって結果が影響するかもしれない - 指摘はいくつかあるが，Student Paperとしては良い -- 会議での議論によって利益を得る（ことができる内容だと思う） #comment
