![[PukiWiki]](image/wadanyan.png "[PukiWiki]")
[[大平/査読結果]] *MSR2014 Reviews [#n18b490d] Dear Masao, Thanks for your submission to the 11th Working Conference on Mining Software Repositories (MSR 2014). We regret to inform you that your paper, Optimizing Task Assignments in the Bug Triaging Process: 0-1 Integer Programming for Large-Scale Projects, has not been accepted for inclusion in the conference program. We received a record number of 112 submissions (85 full and 27 short research and practice paper submissions); out of the 85 full paper submissions, we accepted 29 papers, and out of the 27 short paper submissions, we accepted 10 papers. These are acceptance rates of 34% for full papers and 37% for short papers. Every paper received at least three reviews by members of the PC, and was carefully discussed by PC members until a consensus was reached. We personally examined all of the reviews and online discussion when making the decisions. All decisions were only based on the quality of the papers and on the outcome of the discussions. We did not target any minimum nor maximum number of papers to be accepted. MSR targeted solid research contributions. There are many promising research avenues described in rejected papers, and in many cases the PC just felt the ideas needed further development before they would be accepted. We enclose below the reviewers' comments to your submission, which we hope you will find useful for your future research. We hope that you will be able to attend the MSR conference in Hyderabad, India, on May 31 - June 1, 2014. Sincerely, Sunghun Kim and Martin Pinzger MSR 2014 Program Co-chairs. **----------------------- REVIEW 1 --------------------- [#vf8e3fd8] The paper proposes to use linear integer programming (LP) to balance the workload of maintainers and thus reduce the time to fix a bug. The workload assignment problem is actually modelled as a knapsack problem; better, each maintained is actually thought of as a servant with finite capacity/queue. Incidentally, bug triaging is not done by linear programming but by an external SVM classifier placed in front of the LP task. Pros: - a nice formulation - interesting results a nice formulation interesting results Cons: the way in which time to fix is computed is obscure it compares apples with oranges; there are 2 tasks 1) assigning the bug to the right - the way in which time to fix is computed is obscure - it compares apples with oranges; there are 2 tasks 1) assigning the bug to the right developer and 2) balancing the workload the problem is formulated as a static problem though there is a continuous stream of bugs - the problem is formulated as a static problem though there is a continuous stream of bugs arriving The paper is easy to read though the English may need improvement; but in any case is not impossible to understand the message. The biggest problem of the paper is that it is unclear how the bug fixing time is evaluated and thus how the maintainer workload is estimated. This element is critical as it impacts results and claims but the authors are not clearly say what they did or what they suggest should be done. In any realistic setting, the time elapsed between opening and changing to bug issue to status FIXED is not realistic. See for example the bug (reported but the authors) that lasted 735 days. If indeed the time is not measured or relevant and as it seems the developer is limited in the number of bug assigned (parameter L) then I have hard time to see where is the difference with the work: M. D. Penta, G. Casazza, G. Antoniol, and E. Merlo, “Modeling web maintenance centers through queue models,” in Csmr, 2001, pp. 131-138. Bottom line, in the above work authors proposed a dynamic system and a very efficient one and by plugging the SVM component as a dispatcher one would obtain a quite interesting system. A similar task assignment problem has given a statical solution in: M. D. Penta, M. Harman, and G. Antoniol, “The use of search-based optimization techniques to schedule and staff software projects: an approach and an empirical study,” Softw., pract. exper., vol. 41, iss. 5, pp. 495-519, 2011. A second serious criticism is that comparing bug triaging with load balancing is an unfair comparison. Of course an SVM (or any other classifier) is not well suited to balance developers workload, it is not a classifier task. To this reader in your reference [1] they were not aiming at balancing the workload or reducing the time to serve as in the above papers. ----------------------- - 問題点１：不具合修正時間の計算方法が不明 -- &new{2014-05-03 (土) 14:08:14}; - 問題点２：開発者の作業負荷の計算方法が不明 -- &new{2014-05-03 (土) 14:08:44}; - OpenからFIXEDまでの時間を使っているのが現実的ではない． -- &new{2014-05-03 (土) 14:23:26}; - 負荷分散とタスク割当ての問題を比較するのは妥当ではない． -- &new{2014-05-03 (土) 14:49:45}; #comment() **----------------------- REVIEW 2 --------------------- [#hd0822cf] The paper presents an study on two Eclipse projects, for determining how automated bug-triaging mechanisms could affect the mean time-to-fix, and a technique for improving bug triaging so that overworked developers are not assigned too many bugs to fix. The paper addresses an interesting and important problem. It is well written, and is easy to understand. Most of if presents a sound methodology. However, it also shows some problems, that are mentioned below. The paper is based on the assumption that good bug triaging algorithms could assign too many bugs to fix to a small set of talented developers. However, there is little discussion of a naive approach to solve the problem: let the result of the bug triaging algorithm just an indication of who could fix the bug. That way, overworked developers will take bugs to fix until they can, while others could go on with the rest. A simple extension of this would be having list of asignees, instead of just one asignee per bug report. In fact, this is not that different of how real projects work, where every developer, using the list of bug reports assigned, but also other pieces of information, decides in which to work. maybe the paper could discuss a bit on how the proposed approach would compare to this one, and to the usual practices of real projects. One point that could be weak in the study on how SVM-based assignments work, is how time is calculated. It seems that for each bug, time-to-fix is calculated (it is not clear if it is time from bug reporting or from bug triaging). But it is not clear that this time is accumulative. That is, a part of that time (maybe a long part) is not actually being "consumed" by the fixer, who is doing other activities meanwhile. So, the discussion on 2.3 about how times are calculated and "assigned" to developers fixing bugs should be validated somehow... For example, a developer could take 6 days to fix a certain bug, because at that time he was very busy with some other activity, but three days later, he could have fixed the same bug in just .5 days because he was free of those other tasks. In fact, at that point, he could have fixed three bugs in 1.5 days... In addition, it is very difficult to estimate how long would it take to fix a bug to a developer who did not actually fix it. Therefore, the way this is decided, should be validated, or is not much more than a wild guess. Since the rest of the paper, in some sense relies on the results derived from this accounting, the paper would benefit from a more detailed discussion, and if possible validation, of these details. In particular, there is one aspect which seems to be very important for optimizing bug assignment, which is minimizing the number of bugs assigned to the same person at the same time. But this aspect does not seem to be taken into account in the solution proposed. In fact, while that solution is optimizing the time-to-fix as it is calculated, it could very well be the case that it assigns many bugs to the same person at the same time. In other words, the results of the approach proposed in section 4 are consistent and interesting given the way the calculus is done, and how time-to-fix for a certain mapping of fixers to bugs is estimated. But there is little evidence that such a mapping, and such a calculus really is similar to what would have happened in reality, if this algorithm had been used for triaging. Another threat to validity that is not mentioned is that the training set is a collection of bugs from 2004 to 2008, while the testing set runs from 2008 to 2012. In those years, the team of developers working in the project could have changed significantly. That could affect the results of the approach, since it somehow considers that the project behaves the same during the training period than during the testing period. Page 2: "one of the automated bug triaging approach" -> "one of the automated bug triaging approaches" Page 3: "and a large number of non-active developers." Maybe "non-active" is not a good term, since (according to the definition) they are really active, although much less active than "active developers". ----------------------- - 論文の想定が現実的なものなのかちょっと議論が必要なんじゃない？→時系列を考慮する -- &new{2014-05-03 (土) 14:48:20}; - 問題１：SVMの割当て方法と時間の計算方法が分からない -- &new{2014-05-03 (土) 14:52:51}; - バグが報告されてからの時間なのか，割当てられてからの時間なのか？ -- &new{2014-05-03 (土) 14:53:40}; - 問題２：修正時間の計算方法が妥当かどうかを議論すべき -- &new{2014-05-03 (土) 14:55:57}; - 学習データとテストデータで登場している開発者は同じなのか？全然違うなら意味が変わってくるんじゃないの？ -- &new{2014-05-03 (土) 14:58:34}; #comment() **----------------------- REVIEW 3 --------------------- [#l9fa9036] Most bug triaging approaches are not cost-aware. This causes a problem as most bugs would then be assigned to a few developers potentially causing these developers to be overwhelmed. To address this problem, a new cost-aware bug triaging approach is proposed. The approach uses 0-1 integer programming to assign bug reports to developers. The integer programming considers an upper bound that determines the amount of bugs that can be assigned to a developer. Experiments have been performed on bug reports from the platform and JDT sub-projects of Eclipse. The experiment results show that the proposed approach can outperform the first work on bug triaging by Anvik et al. [1] and real bug assignment in terms of average time needed to fix a bug. Like: + Interesting usage of integer linear programming to solve cost-aware bug triaging. + Two datasets, from platform and JDT sub-projects of Eclipse, are used to evaluate the proposed approach. + Experiments show that the proposed approach can improve the time it takes for bug reports to get resolved by up to 42.6% as compared to current bug triaging practices and by up to 64.5% compared to the approach proposed by Anvik et al. in 2006. Negative Points: + There exists another approach that is also cost-aware, i.e., reference [22]. Thus there is a need to compare the proposed approach with [22] to demonstrate that it has improved over the state-of-the art. + The baseline approach that is used is rather old -- i.e., reference [1] published in 2006 (8 years ago). There are many more recent approaches, e.g., reference [9,27] which are published last year. There is a need to compare to more recent approaches to demonstrate that the proposed approach has outperformed the state-of-the-art. + There is a need to compare with a trivial baseline: Given a bug report of component C, recommend a developer which has the lowest bug fixing time for component C. If my understanding of the evaluation approach is correct, this baseline would outperform the proposed approach. + The following assumption used to evaluate the proposed approach needs to be validated empirically: -- Assumption: The bug fixing time for a bug of a particular component c by a particular developer d, can be accurately estimated by the average time d fixes bugs for component c. [#m0288cf2] -- Concerns: Two bugs can belong to the same component, but one is of high difficulty and another is of low difficulty. Also, one bug could be of high importance (and thus get fixed ASAP), while another is of low importance. Would these variations affect bug fix time of individual bugs? Furthermore, some developers only fixed 1 bug from a particular component, while some others fixed 1000 bugs from a particular component. The estimate (using average fix time) for the first group of developers would be much poorer than the estimate for the second group of developers. [#r4eeb46e] -- Assumption: Assigning most of the bugs to a few developers would make these developers overwhelmed. [#o5fd3802] -- Concerns: The fact is most bug reports are fixed by a small group of "active developers"[19]. It is unclear why this is the case? Could it be possible that these active developers devote much of their time (voluntarily) to open source development while the other developers only devote little of their time to open source development? If this is the case, then assigning 10 bugs to developers that spend 10 hours a day would be lighter than assigning 1 bug to developers that spend 10 hours a month. [#nbed2981] + The value of L affects the number of bugs that are successfully assigned and the time needed to resolve the bugs. To make the comparison fair, I think there is a need to set the value of L such that the number of bugs that are successfully assigned by the proposed approach is the same with those successfully assigned by SVM or real assignments. + The constraints used for integer programming seems a bit wrong to me. In particular there is no constraint that specifies that a single bug can only be assigned to one developer. + Effectively only bug reports from one system is analysed: Eclipse. There is a need to analyse more systems. + Are the differences in the time needed to fix a bug significant? A statistical test can be used to answer this question. + During assignment, does the proposed approach take into consideration whether a previously assigned bug report has been completed by X%? Minor: + Section 3.1 can be extended to describe integer programming better. The current description requires readers to understand integer programming to appreciate it. + The description of time limitation (L) can be improved. In particular, I don't quite understand: "The value of P_{ij} depends on the time limitation L. The value of each P_{ij} can always be 5 if L is infinite". There is a need for further explanation. + Many of the related works listed in the paper are rather old and newer papers on similar topics have been published. ----------------------- - 問題１：cost-aware研究との比較がない -- &new{2014-05-03 (土) 16:24:20}; - 問題２：Anvikは古い．最新の研究と比較していない． -- &new{2014-05-03 (土) 16:27:27}; - 問題３：バグ修正のコストが最も低くなる開発者にタスクを割当てる方法の方が良いのでは（問題１に関連？） -- &new{2014-05-03 (土) 16:29:34}; - 問題４−１：平均修正時間を用いるのが妥当なのかを検証するべき -- &new{2014-05-03 (土) 16:30:52}; - 緊急性の高い（重要な）バグとそうでないバグが同じコンポーネントで発見される事もあり得る．バグ修正時間の見積りに影響するのでは． -- &new{2014-05-03 (土) 16:35:15}; - あるコンポーネントのバグを1つ修正した開発者と1000個修正した開発者がいたときに，前者は後者よりも劣ることになるんじゃないの？ -- &new{2014-05-03 (土) 16:36:42}; - 問題４−２：OSS開発の場合，開発者が作業できる時間は人それぞれだから，タスク集中してるってことはその人がそれだけ時間を使えるだけなのでは？ -- &new{2014-05-03 (土) 16:39:03}; - 問題５：Lの値は割当てに成功したバグの数とバグ修正に要した時間に影響するから，既存手法と比較するときには割当てに成功したバグの数を会わせないと意味がないんじゃないの． -- &new{2014-05-03 (土) 16:43:11}; - 問題６：1つのバグが1つの開発者に割当てられるという制約は間違っていると思う． -- &new{2014-05-03 (土) 16:44:27}; - 問題７：Eclipseだけしか分析していない -- &new{2014-05-03 (土) 16:44:53}; - 問題８：1つのバグを修正するのに要する時間に有意差はあるんか？ -- &new{2014-05-03 (土) 16:46:17}; - 問題９：割当て時に修正集の不具合が何割完了しているかを考慮しているのか？ -- &new{2014-05-03 (土) 16:47:56}; - IPについてもっとちゃんとかけ -- [[Minor]] &new{2014-05-03 (土) 16:48:49}; - Lについて理解できん -- [[Minor]] &new{2014-05-03 (土) 16:50:07}; - 最新の関連研究がもっとあるだろ -- &new{2014-05-03 (土) 16:50:39}; #comment()
