![[PukiWiki]](image/wadanyan.png "[PukiWiki]")
[[大平/査読結果]] *ICSME2014 Reviews [#ne2122c1] Dear Masao, We regret to inform you that we are unable to include your submission, Bug Triaging as An Optimization Problem: 0-1 Integer Programming for Mitigating Imbalanced Task Assignments in Large-Scale OSS Development, in the Research Track of ICSME 2014. In total, we received 267 abstracts and 210 full paper submissions. Only 40 of these papers were accepted, yielding an acceptance rate of 19%. All papers went through a rigorous reviewing process in which they received at least three independent reviews, followed by discussions among the assigned reviewers, and, finally, a discussion open to the entire program committee. The program committee selected papers for inclusion in the program without targeting any quota or acceptance rate: merits and weaknesses of each paper were discussed independently of other papers taken into consideration. Attached to this email, you will find the reviews of your paper. We hope that you will find them helpful in the revision of your paper for other ICSME tracks or other venues. In particular, we hope that you will consider submitting your work to some of the other tracks of ICSME 2014: the Early Research Achievement (ERA) Track, the Industry Track, the Tool Demo Track, and the Doctoral Symposium; especially if the reviewers suggest you to do so. Note that the deadline for abstract submissions for these tracks is June 23 (AoE), the deadline for full submissions (4 pages) is June 30 (with grace period til July 7). More information on these can be found at http://www.icsme.org/call-for-papers You may also consider submitting your work to one of the workshops co-located with ICSME 2014. More information on those can be found at http://www.icsme.org/2014 We sincerely hope that you will attend ICSME 2014 in Victoria and look forward to seeing you there September 28 - October 3, 2014. Thank you for submitting your paper to ICSME 2014 Best regards, Leon Moonen and Lori Pollock ICSME 2014 Program Co-Chairs **----------------------- REVIEW 1 --------------------- [#y9f436b9] This paper proposes task assignment optimization for OSS projects.Combining integer programming and SVM machine learning method, the authors propose an assignment algorithm with workload limit for each developer. The proposed algorithm and traditional SVM approach have been investigated with two projects, Firefox and Eclipse, and the proposed approach shows better performance of the balanced assignment and total bug fix time. + Overall, i enjoyed reading this paper. The paper proposes a new practical method of task allocation ＋ Overall, i enjoyed reading this paper. The paper proposes a new practical method of task allocation for the OSS projects, where small set of developers tends to get many tasks.This approach clearly sets the limitation of the total task load, and the solution would follow this constraint. Also, it shows the reduction of the total bug-fix time with this new algorithm. + An empirical validation has been performed with two large data sets, Foirefox and Eclipse, and ＋ An empirical validation has been performed with two large data sets, Foirefox and Eclipse, and the results clearly show the advantages of the proposed approach. 以下マイナスポイント - It's not clear if the proposed approach would work interactively or not. I would guess that the system gets the bug-fix allocation history and the new bug cost, and it answers the best-fit allocation for that new bug. Or It would compute the optimal allocation for the whole bug data of the target systems at the same time. In the latter case, practical use of this approach might be difficult. Authors should clearly describe the proposed approach in such sense.Fig.1 and Fig.3 do not help to understand it. - Explanation of the model and formula is not sufficient. For example, formula (1)~(6) give only limited overview of the model, so simple example of those formula would help to understand more in depth. - In the experiment, the authors have used a previous approach. However, it is not clear that the previous approach would be constructed with sufficient consideration of balanced assignment. I would guess that the older approach had different perspective and target, rather than balanced assignment. Authors should clearly present this issue. - Explanation of Fig.1 does not fit to the elements of Fig.1. Maybe more clear explanation would be needed for Fig.1. - Meaning of "common" is not clear. - For Table VI, we would guess that the difference of SVM and IP is the advantage of this approach. Is the difference significant? I'm not sure we would do such statistic in this case, but the authors would mention on this issue. - It seems that the approach assumes each bug is independent; no relations or no dependence. But such assumption is unrealistic I believe. It is natural such relations/dependence may raise/reduce the cost of each bug. - What is the actual "objective functions?" It is clearly defined in Section 2,but it would be a tough work to define preferences and objective variables (and also define their values of each developer). Are there any examples of preferences/objectives? - As time goes by, time limitation L might be changed (and also it depends on each developer).The approach employs L is constant integer; is it safe assumption for Firefox/Eclipse case? - As the authors described in Section 7, it would be interesting to apply the approach to individuals, and it would be a chance to do a experiment with some subjects. ------------------------------------------------------------- #comment() **----------------------- REVIEW 2 --------------------- [#p92c1b42] The paper presents an study on two Eclipse projects, for determining how automated bug-triaging mechanisms could affect the mean time-to-fix, and a technique for improving bug triaging so that overworked developers are not assigned too many bugs to fix. Summary: The paper addresses an interesting and important problem. It is well written, and is easy to understand. Most of if presents a sound methodology. However, it also shows some problems, that are mentioned below. In this paper, the authors formulate bug triaging as integer programming, an optimization problem. That is, given a set of developers and a set of bugs need fixing, bug triaging is to assign each task to a developer, such that the assignment maximizes the total "preference" of developers over bugs while keeping the total "fixing time" of each developer within his/her time limit. The "preference" is computed by a SVM-based method proposed in prior work and the "fixing time" is computed as the average fixing time of each developer on bugs of the same component previously fixed by him/her. The paper is based on the assumption that good bug triaging algorithms could assign too many bugs to fix to a small set of talented developers. However, there is little discussion of a naive approach to solve the problem: let the result of the bug triaging algorithm just an indication of who could fix the bug. That way, overworked developers will take bugs to fix until they can, while others could go on with the rest. A simple extension of this would be having list of asignees, instead of just one asignee per bug report. In fact, this is not that different of how real projects work, where every developer, using the list of bug reports assigned, but also other pieces of information, decides in which to work. maybe the paper could discuss a bit on how the proposed approach would compare to this one, and to the usual practices of real projects. Comment: One point that could be weak in the study on how SVM-based assignments work, is how time is calculated. It seems that for each bug, time-to-fix is calculated (it is not clear if it is time from bug reporting or from bug triaging). But it is not clear that this time is accumulative. That is, a part of that time (maybe a long part) is not actually being "consumed" by the fixer, who is doing other activities meanwhile. So, the discussion on 2.3 about how times are calculated and "assigned" to developers fixing bugs should be validated somehow... For example, a developer could take 6 days to fix a certain bug, because at that time he was very busy with some other activity, but three days later, he could have fixed the same bug in just .5 days because he was free of those other tasks. In fact, at that point, he could have fixed three bugs in 1.5 days... Overall, the authors propose a novel approach for an important problem of optimal bug assignment, with multiple developers and multiple bugs. The paper is well-written and easy to understand. However, I have some concerns with the proposed problem and solution. In addition, it is very difficult to estimate how long would it take to fix a bug to a developer who did not actually fix it. Therefore, the way this is decided, should be validated, or is not much more than a wild guess. 1. I wonder why the optimal bug triaging needs to be formulated as integer programming (IP) and why fixing time (C_ij) and time limit (L) needs to be integers? As far as I know, IP is NP-hard and 0-1 IP (as in this case) is NP-complete, thus, are very computational expensive. The authors report that "in our experimental setting, a calculation only took about two minutes for the two projects." This fast performance might be due to the small size of experiment data (e.g. just around 20 developers (Table V) and 40-80 bugs/month (Figure 5)). However, software projects might have thousands of developers and receive hundred bugs each day. In such case, IP would be exponentially more computational expensive and thus, is not practical. Since the rest of the paper, in some sense relies on the results derived from this accounting, the paper would benefit from a more detailed discussion, and if possible validation, of these details. 2. Why the goal of bug triaging is to maximize the total "preference"? If we want to reduce fixing time and avoid over-assignment, why don't we try to minimize the total fixing time, or some cost/utility functions involving the allocated tasks (e.g. the unbalance of task assignments between developers). More seriously, the "preference" here is *predicted* by the SVM-based model, which has prediction errors and cannot be measured or quantitatively validated. Summing "preference" is also troublesome to me, as I cannot interpret the practical meaning of this "total preference". (The preference score can be used to rank developers for an individual bug, but I don't think we can add "preference" of two different bugs). In particular, there is one aspect which seems to be very important for optimizing bug assignment, which is minimizing the number of bugs assigned to the same person at the same time. But this aspect does not seem to be taken into account in the solution proposed. In fact, while that solution is optimizing the time-to-fix as it is calculated, it could very well be the case that it assigns many bugs to the same person at the same time. 3. The proposed formulation does not take into account the time and semantic dependency between bugs. For example, a bug B needs to be fixed before a bug B' can be fixed, while B' might even be filed long before B. Thus, although B' is filed very soon, it should not be assigned right away. In other words, the results of the approach proposed in section 4 are consistent and interesting given the way the calculus is done, and how time-to-fix for a certain mapping of fixers to bugs is estimated. But there is little evidence that such a mapping, and such a calculus really is similar to what would have happened in reality, if this algorithm had been used for triaging. 4. The computation of fixing time of a developer on a bug is tricky. First, a developer might be assigned several bugs at the same time. He/she might be also assigned to several other tasks. Thus, it is difficult to measure the actual time he/she spends for a particular bug. The authors do not explain in details how they determine the fixing time in their experiment. Another threat to validity that is not mentioned is that the training set is a collection of bugs from 2004 to 2008, while the testing set runs from 2008 to 2012. In those years, the team of developers working in the project could have changed significantly. That could affect the results of the approach, since it somehow considers that the project behaves the same during the training period than during the testing period. 5. Predicting fixing time of a developer for a bug (C_ij) as "average time to fix bugs which are categorized into the same component" would be pretty inaccurate. As seen in Table III, fixing time varies significantly (e.g. from 1 day to 40 days). Thus, the average time (either mean or median) will be a poor estimate of the actual fixing time. This predicting method also makes the proposed approach less robust, as it cannot assign bugs to developers who have not fixed relevant bugs before. Page 2: "one of the automated bug triaging approach" -> "one of the automated bug triaging approaches" 6. The experiment setting has some flaws. For example, bugs should be sorted by filing/fixing time and bugs in training set should be filed/fixed before bugs in testing set. Page 3: "and a large number of non-active developers." Maybe "non-active" is not a good term, since (according to the definition) they are really active, although much less active than "active developers". ----------------------- - 論文の想定が現実的なものなのかちょっと議論が必要なんじゃない？→時系列を考慮する -- &new{2014-05-03 (土) 14:48:20}; - 問題１：SVMの割当て方法と時間の計算方法が分からない -- &new{2014-05-03 (土) 14:52:51}; - バグが報告されてからの時間なのか，割当てられてからの時間なのか？ -- &new{2014-05-03 (土) 14:53:40}; - 問題２：修正時間の計算方法が妥当かどうかを議論すべき -- &new{2014-05-03 (土) 14:55:57}; - 学習データとテストデータで登場している開発者は同じなのか？全然違うなら意味が変わってくるんじゃないの？ -- &new{2014-05-03 (土) 14:58:34}; ------------------------------------------------------------- #comment() **----------------------- REVIEW 3 --------------------- [#v48c54f7] Most bug triaging approaches are not cost-aware. This causes a problem as most bugs would then be assigned to a few developers potentially causing these developers to be overwhelmed. To address this problem, a new cost-aware bug triaging approach is proposed. The approach uses 0-1 integer programming to assign bug reports to developers. The integer programming considers an upper bound that determines the amount of bugs that can be assigned to a developer. Experiments have been performed on bug reports from the platform and JDT sub-projects of Eclipse. The experiment results show that the proposed approach can outperform the first work on bug triaging by Anvik et al. [1] and real bug assignment in terms of average time needed to fix a bug. This paper proposed a new capacity-aware bug triaging technique using the combination of 0-1 integer programing and SVM. It assigns bug fixing tasks in consideration of a developer's workload capacity. The authors formulated the assignment task as a 0-1 integer programming problem. The simulation-based evaluation was performed using two popular open source projects, and the results shows that days to fix were reduced and developers' loads were balanced. Like: + Interesting usage of integer linear programming to solve cost-aware bug triaging. + Two datasets, from platform and JDT sub-projects of Eclipse, are used to evaluate the proposed approach. + Experiments show that the proposed approach can improve the time it takes for bug reports to get resolved by up to 42.6% as compared to current bug triaging practices and by up to 64.5% compared to the approach proposed by Anvik et al. in 2006. This paper is well-structured and the formulation to a 0-1 IP is interesting. The evaluation is not complete because it is simulation-based and its data set is not so large; however, the careful analysis and discussion can successfully support the conclusions. Though more evaluations are needed and there are many future work issues, this study is promising. Negative Points: + There exists another approach that is also cost-aware, i.e., reference [22]. Thus there is a need to compare the proposed approach with [22] to demonstrate that it has improved over the state-of-the art. + The baseline approach that is used is rather old -- i.e., reference [1] published in 2006 (8 years ago). There are many more recent approaches, e.g., reference [9,27] which are published last year. There is a need to compare to more recent approaches to demonstrate that the proposed approach has outperformed the state-of-the-art. + There is a need to compare with a trivial baseline: Given a bug report of component C, recommend a developer which has the lowest bug fixing time for component C. If my understanding of the evaluation approach is correct, this baseline would outperform the proposed approach. + The following assumption used to evaluate the proposed approach needs to be validated empirically: -- Assumption: The bug fixing time for a bug of a particular component c by a particular developer d, can be accurately estimated by the average time d fixes bugs for component c. [#m0288cf2] -- Concerns: Two bugs can belong to the same component, but one is of high difficulty and another is of low difficulty. Also, one bug could be of high importance (and thus get fixed ASAP), while another is of low importance. Would these variations affect bug fix time of individual bugs? Furthermore, some developers only fixed 1 bug from a particular component, while some others fixed 1000 bugs from a particular component. The estimate (using average fix time) for the first group of developers would be much poorer than the estimate for the second group of developers. [#r4eeb46e] -- Assumption: Assigning most of the bugs to a few developers would make these developers overwhelmed. [#o5fd3802] -- Concerns: The fact is most bug reports are fixed by a small group of "active developers"[19]. It is unclear why this is the case? Could it be possible that these active developers devote much of their time (voluntarily) to open source development while the other developers only devote little of their time to open source development? If this is the case, then assigning 10 bugs to developers that spend 10 hours a day would be lighter than assigning 1 bug to developers that spend 10 hours a month. [#nbed2981] + The value of L affects the number of bugs that are successfully assigned and the time needed to resolve the bugs. To make the comparison fair, I think there is a need to set the value of L such that the number of bugs that are successfully assigned by the proposed approach is the same with those successfully assigned by SVM or real assignments. + The constraints used for integer programming seems a bit wrong to me. In particular there is no constraint that specifies that a single bug can only be assigned to one developer. + Effectively only bug reports from one system is analysed: Eclipse. There is a need to analyse more systems. + Are the differences in the time needed to fix a bug significant? A statistical test can be used to answer this question. + During assignment, does the proposed approach take into consideration whether a previously assigned bug report has been completed by X%? To improve the paper, please consider the following points: Minor: + Section 3.1 can be extended to describe integer programming better. The current description requires readers to understand integer programming to appreciate it. + The description of time limitation (L) can be improved. In particular, I don't quite understand: "The value of P_{ij} depends on the time limitation L. The value of each P_{ij} can always be 5 if L is infinite". There is a need for further explanation. + Many of the related works listed in the paper are rather old and newer papers on similar topics have been published. Since 0-1 IP problems are NP-hard, the scalability of the execution time of this technique should be discussed. The authors already wrote "two minute for the two project" in Sec.III; however, more general and deep discussion would help readers. ----------------------- An example of Preference Pij and the intent of Exp.5 is not presented. Besides, the definition of Pij is absent in the paper. Since [2] does not describe how to calculate such score explicitly, its minimal explanation should be described here. - 問題１：cost-aware研究との比較がない -- &new{2014-05-03 (土) 16:24:20}; - 問題２：Anvikは古い．最新の研究と比較していない． -- &new{2014-05-03 (土) 16:27:27}; - 問題３：バグ修正のコストが最も低くなる開発者にタスクを割当てる方法の方が良いのでは（問題１に関連？） -- &new{2014-05-03 (土) 16:29:34}; - 問題４−１：平均修正時間を用いるのが妥当なのかを検証するべき -- &new{2014-05-03 (土) 16:30:52}; - 緊急性の高い（重要な）バグとそうでないバグが同じコンポーネントで発見される事もあり得る．バグ修正時間の見積りに影響するのでは． -- &new{2014-05-03 (土) 16:35:15}; - あるコンポーネントのバグを1つ修正した開発者と1000個修正した開発者がいたときに，前者は後者よりも劣ることになるんじゃないの？ -- &new{2014-05-03 (土) 16:36:42}; - 問題４−２：OSS開発の場合，開発者が作業できる時間は人それぞれだから，タスク集中してるってことはその人がそれだけ時間を使えるだけなのでは？ -- &new{2014-05-03 (土) 16:39:03}; - 問題５：Lの値は割当てに成功したバグの数とバグ修正に要した時間に影響するから，既存手法と比較するときには割当てに成功したバグの数を会わせないと意味がないんじゃないの． -- &new{2014-05-03 (土) 16:43:11}; - 問題６：1つのバグが1つの開発者に割当てられるという制約は間違っていると思う． -- &new{2014-05-03 (土) 16:44:27}; - 問題７：Eclipseだけしか分析していない -- &new{2014-05-03 (土) 16:44:53}; - 問題８：1つのバグを修正するのに要する時間に有意差はあるんか？ -- &new{2014-05-03 (土) 16:46:17}; - 問題９：割当て時に修正集の不具合が何割完了しているかを考慮しているのか？ -- &new{2014-05-03 (土) 16:47:56}; - IPについてもっとちゃんとかけ -- [[Minor]] &new{2014-05-03 (土) 16:48:49}; - Lについて理解できん -- [[Minor]] &new{2014-05-03 (土) 16:50:07}; - 最新の関連研究がもっとあるだろ -- &new{2014-05-03 (土) 16:50:39}; MINOR ISSUES: - Figure 6, 7 and 8 are too small to read. - Figure 2, 4, and 5 have no Y-axis labels. - The explanation of "common" at Sec.IV-A is hard to understand and needs more words. - The calculation of Cij is ambiguous and needs a more concrete description. - In p.3, "our our" -> "our" - At the 4th paragraph in Sec.IV-B, the reason should be described briefly, and an indentation is missing. - In Table VI, "the less period" -> "the less busy period" - The first paragraph in Sec.III-B 6) is unclear. L is not the parameter for Pij or the objective function but the parameter for this technique. - At the 1st paragraph in Sec.V-A, "a tool large" -> "a too large" - At the 1st paragraph in Sec.V-B, "is is" -> "is" ----------------------- #comment()
