![[PukiWiki]](image/wadanyan.png "[PukiWiki]")
Dear Masao,
Thanks for your submission to the 11th Working Conference on Mining Software Repositories (MSR 2014). We regret to inform you that your paper, Optimizing Task Assignments in the Bug Triaging Process: 0-1 Integer Programming for Large-Scale Projects, has not been accepted for inclusion in the conference program.
We received a record number of 112 submissions (85 full and 27 short research and practice paper submissions); out of the 85 full paper submissions, we accepted 29 papers, and out of the 27 short paper submissions, we accepted 10 papers. These are acceptance rates of 34% for full papers and 37% for short papers. Every paper received at least three reviews by members of the PC, and was carefully discussed by PC members until a consensus was reached. We personally examined all of the reviews and online discussion when making the decisions. All decisions were only based on the quality of the papers and on the outcome of the discussions. We did not target any minimum nor maximum number of papers to be accepted.
MSR targeted solid research contributions. There are many promising research avenues described in rejected papers, and in many cases the PC just felt the ideas needed further development before they would be accepted. We enclose below the reviewers' comments to your submission, which we hope you will find useful for your future research. We hope that you will be able to attend the MSR conference in Hyderabad, India, on May 31 - June 1, 2014.
Sincerely,
Sunghun Kim and Martin Pinzger MSR 2014 Program Co-chairs.
The paper proposes to use linear integer programming (LP) to balance the workload of maintainers and thus reduce the time to fix a bug. The workload assignment problem is actually modelled as a knapsack problem; better, each maintained is actually thought of as a servant with finite capacity/queue. Incidentally, bug triaging is not done by linear programming but by an external SVM classifier placed in front of the LP task.
Pros:
a nice formulation
interesting results
Cons:
the way in which time to fix is computed is obscure
it compares apples with oranges; there are 2 tasks 1) assigning the bug to the right developer and 2) balancing the workload
the problem is formulated as a static problem though there is a continuous stream of bugs arriving
The paper is easy to read though the English may need improvement; but in any case is not impossible to understand the message. The biggest problem of the paper is that it is unclear how the bug fixing time is evaluated and thus how the maintainer workload is estimated. This element is critical as it impacts results and claims but the authors are not clearly say what they did or what they suggest should be done. In any realistic setting, the time elapsed between opening and changing to bug issue to status FIXED is not realistic. See for example the bug (reported but the authors) that lasted 735 days. If indeed the time is not measured or relevant and as it seems the developer is limited in the number of bug assigned (parameter L) then I have hard time to see where is the difference with the work:
M. D. Penta, G. Casazza, G. Antoniol, and E. Merlo, “Modeling web maintenance centers through queue models,” in Csmr, 2001, pp. 131-138.
Bottom line, in the above work authors proposed a dynamic system and a very efficient one and by plugging the SVM component as a dispatcher one would obtain a quite interesting system. A similar task assignment problem has given a statical solution in:
M. D. Penta, M. Harman, and G. Antoniol, “The use of search-based optimization techniques to schedule and staff software projects: an approach and an empirical study,” Softw., pract. exper., vol. 41, iss. 5, pp. 495-519, 2011.
A second serious criticism is that comparing bug triaging with load balancing is an unfair comparison. Of course an SVM (or any other classifier) is not well suited to balance developers workload, it is not a classifier task. To this reader in your reference [1] they were not aiming at balancing the workload or reducing the time to serve as in the above papers.
The paper presents an study on two Eclipse projects, for determining how automated bug-triaging mechanisms could affect the mean time-to-fix, and a technique for improving bug triaging so that overworked developers are not assigned too many bugs to fix.
The paper addresses an interesting and important problem. It is well written, and is easy to understand. Most of if presents a sound methodology. However, it also shows some problems, that are mentioned below.
The paper is based on the assumption that good bug triaging algorithms could assign too many bugs to fix to a small set of talented developers. However, there is little discussion of a naive approach to solve the problem: let the result of the bug triaging algorithm just an indication of who could fix the bug. That way, overworked developers will take bugs to fix until they can, while others could go on with the rest. A simple extension of this would be having list of asignees, instead of just one asignee per bug report. In fact, this is not that different of how real projects work, where every developer, using the list of bug reports assigned, but also other pieces of information, decides in which to work. maybe the paper could discuss a bit on how the proposed approach would compare to this one, and to the usual practices of real projects.
One point that could be weak in the study on how SVM-based assignments work, is how time is calculated. It seems that for each bug, time-to-fix is calculated (it is not clear if it is time from bug reporting or from bug triaging). But it is not clear that this time is accumulative. That is, a part of that time (maybe a long part) is not actually being "consumed" by the fixer, who is doing other activities meanwhile. So, the discussion on 2.3 about how times are calculated and "assigned" to developers fixing bugs should be validated somehow... For example, a developer could take 6 days to fix a certain bug, because at that time he was very busy with some other activity, but three days later, he could have fixed the same bug in just .5 days because he was free of those other tasks. In fact, at that point, he could have fixed three bugs in 1.5 days...
In addition, it is very difficult to estimate how long would it take to fix a bug to a developer who did not actually fix it. Therefore, the way this is decided, should be validated, or is not much more than a wild guess.
Since the rest of the paper, in some sense relies on the results derived from this accounting, the paper would benefit from a more detailed discussion, and if possible validation, of these details.
In particular, there is one aspect which seems to be very important for optimizing bug assignment, which is minimizing the number of bugs assigned to the same person at the same time. But this aspect does not seem to be taken into account in the solution proposed. In fact, while that solution is optimizing the time-to-fix as it is calculated, it could very well be the case that it assigns many bugs to the same person at the same time.
In other words, the results of the approach proposed in section 4 are consistent and interesting given the way the calculus is done, and how time-to-fix for a certain mapping of fixers to bugs is estimated. But there is little evidence that such a mapping, and such a calculus really is similar to what would have happened in reality, if this algorithm had been used for triaging.
Another threat to validity that is not mentioned is that the training set is a collection of bugs from 2004 to 2008, while the testing set runs from 2008 to 2012. In those years, the team of developers working in the project could have changed significantly. That could affect the results of the approach, since it somehow considers that the project behaves the same during the training period than during the testing period.
Page 2: "one of the automated bug triaging approach" -> "one of the automated bug triaging approaches"
Page 3: "and a large number of non-active developers." Maybe "non-active" is not a good term, since (according to the definition) they are really active, although much less active than "active developers".
Most bug triaging approaches are not cost-aware. This causes a problem as most bugs would then be assigned to a few developers potentially causing these developers to be overwhelmed. To address this problem, a new cost-aware bug triaging approach is proposed. The approach uses 0-1 integer programming to assign bug reports to developers. The integer programming considers an upper bound that determines the amount of bugs that can be assigned to a developer. Experiments have been performed on bug reports from the platform and JDT sub-projects of Eclipse. The experiment results show that the proposed approach can outperform the first work on bug triaging by Anvik et al. [1] and real bug assignment in terms of average time needed to fix a bug.
Like:
Negative Points:
Minor:
