大平/査読結果

ICSME2014 Reviews†

Dear Masao,

We regret to inform you that we are unable to include your submission, Bug Triaging as An Optimization Problem: 0-1 Integer Programming for Mitigating Imbalanced Task Assignments in Large-Scale OSS Development, in the Research Track of ICSME 2014. In total, we received 267 abstracts and 210 full paper submissions. Only 40 of these papers were accepted, yielding an acceptance rate of 19%.

All papers went through a rigorous reviewing process in which they received at least three independent reviews, followed by discussions among the assigned reviewers, and, finally, a discussion open to the entire program committee. The program committee selected papers for inclusion in the program without targeting any quota or acceptance rate: merits and weaknesses of each paper were discussed independently of other papers taken into consideration.

Attached to this email, you will find the reviews of your paper. We hope that you will find them helpful in the revision of your paper for other ICSME tracks or other venues. In particular, we hope that you will consider submitting your work to some of the other tracks of ICSME 2014: the Early Research Achievement (ERA) Track, the Industry Track, the Tool Demo Track, and the Doctoral Symposium; especially if the reviewers suggest you to do so. Note that the deadline for abstract submissions for these tracks is June 23 (AoE), the deadline for full submissions (4 pages) is June 30 (with grace period til July 7). More information on these can be found at http://www.icsme.org/call-for-papers

You may also consider submitting your work to one of the workshops co-located with ICSME 2014. More information on those can be found at http://www.icsme.org/2014

We sincerely hope that you will attend ICSME 2014 in Victoria and look forward to seeing you there September 28 - October 3, 2014.

Thank you for submitting your paper to ICSME 2014

Best regards, Leon Moonen and Lori Pollock ICSME 2014 Program Co-Chairs

↑

----------------------- REVIEW 1 ---------------------†

This paper proposes task assignment optimization for OSS projects.Combining integer programming and SVM machine learning method, the authors propose an assignment algorithm with workload limit for each developer. The proposed algorithm and traditional SVM approach have been investigated with two projects, Firefox and Eclipse, and the proposed approach shows better performance of the balanced assignment and total bug fix time.

  1. Overall, i enjoyed reading this paper. The paper proposes a new practical method of task allocation for the OSS projects, where small set of developers tends to get many tasks.This approach clearly sets the limitation of the total task load, and the solution would follow this constraint. Also, it shows the reduction of the total bug-fix time with this new algorithm.
  1. An empirical validation has been performed with two large data sets, Foirefox and Eclipse, and the results clearly show the advantages of the proposed approach.

↑

----------------------- REVIEW 2 ---------------------†

The paper presents an study on two Eclipse projects, for determining how automated bug-triaging mechanisms could affect the mean time-to-fix, and a technique for improving bug triaging so that overworked developers are not assigned too many bugs to fix.

The paper addresses an interesting and important problem. It is well written, and is easy to understand. Most of if presents a sound methodology. However, it also shows some problems, that are mentioned below.

The paper is based on the assumption that good bug triaging algorithms could assign too many bugs to fix to a small set of talented developers. However, there is little discussion of a naive approach to solve the problem: let the result of the bug triaging algorithm just an indication of who could fix the bug. That way, overworked developers will take bugs to fix until they can, while others could go on with the rest. A simple extension of this would be having list of asignees, instead of just one asignee per bug report. In fact, this is not that different of how real projects work, where every developer, using the list of bug reports assigned, but also other pieces of information, decides in which to work. maybe the paper could discuss a bit on how the proposed approach would compare to this one, and to the usual practices of real projects.

One point that could be weak in the study on how SVM-based assignments work, is how time is calculated. It seems that for each bug, time-to-fix is calculated (it is not clear if it is time from bug reporting or from bug triaging). But it is not clear that this time is accumulative. That is, a part of that time (maybe a long part) is not actually being "consumed" by the fixer, who is doing other activities meanwhile. So, the discussion on 2.3 about how times are calculated and "assigned" to developers fixing bugs should be validated somehow... For example, a developer could take 6 days to fix a certain bug, because at that time he was very busy with some other activity, but three days later, he could have fixed the same bug in just .5 days because he was free of those other tasks. In fact, at that point, he could have fixed three bugs in 1.5 days...

In addition, it is very difficult to estimate how long would it take to fix a bug to a developer who did not actually fix it. Therefore, the way this is decided, should be validated, or is not much more than a wild guess.

Since the rest of the paper, in some sense relies on the results derived from this accounting, the paper would benefit from a more detailed discussion, and if possible validation, of these details.

In particular, there is one aspect which seems to be very important for optimizing bug assignment, which is minimizing the number of bugs assigned to the same person at the same time. But this aspect does not seem to be taken into account in the solution proposed. In fact, while that solution is optimizing the time-to-fix as it is calculated, it could very well be the case that it assigns many bugs to the same person at the same time.

In other words, the results of the approach proposed in section 4 are consistent and interesting given the way the calculus is done, and how time-to-fix for a certain mapping of fixers to bugs is estimated. But there is little evidence that such a mapping, and such a calculus really is similar to what would have happened in reality, if this algorithm had been used for triaging.

Another threat to validity that is not mentioned is that the training set is a collection of bugs from 2004 to 2008, while the testing set runs from 2008 to 2012. In those years, the team of developers working in the project could have changed significantly. That could affect the results of the approach, since it somehow considers that the project behaves the same during the training period than during the testing period.

Page 2: "one of the automated bug triaging approach" -> "one of the automated bug triaging approaches"

Page 3: "and a large number of non-active developers." Maybe "non-active" is not a good term, since (according to the definition) they are really active, although much less active than "active developers".


↑

----------------------- REVIEW 3 ---------------------†

Most bug triaging approaches are not cost-aware. This causes a problem as most bugs would then be assigned to a few developers potentially causing these developers to be overwhelmed. To address this problem, a new cost-aware bug triaging approach is proposed. The approach uses 0-1 integer programming to assign bug reports to developers. The integer programming considers an upper bound that determines the amount of bugs that can be assigned to a developer. Experiments have been performed on bug reports from the platform and JDT sub-projects of Eclipse. The experiment results show that the proposed approach can outperform the first work on bug triaging by Anvik et al. [1] and real bug assignment in terms of average time needed to fix a bug.

Like:

  1. Interesting usage of integer linear programming to solve cost-aware bug triaging.
  2. Two datasets, from platform and JDT sub-projects of Eclipse, are used to evaluate the proposed approach.
  3. Experiments show that the proposed approach can improve the time it takes for bug reports to get resolved by up to 42.6% as compared to current bug triaging practices and by up to 64.5% compared to the approach proposed by Anvik et al. in 2006.

Negative Points:

  1. There exists another approach that is also cost-aware, i.e., reference [22]. Thus there is a need to compare the proposed approach with [22] to demonstrate that it has improved over the state-of-the art.
  2. The baseline approach that is used is rather old -- i.e., reference [1] published in 2006 (8 years ago). There are many more recent approaches, e.g., reference [9,27] which are published last year. There is a need to compare to more recent approaches to demonstrate that the proposed approach has outperformed the state-of-the-art.
  3. There is a need to compare with a trivial baseline: Given a bug report of component C, recommend a developer which has the lowest bug fixing time for component C. If my understanding of the evaluation approach is correct, this baseline would outperform the proposed approach.
  4. The following assumption used to evaluate the proposed approach needs to be validated empirically:
    • Assumption: The bug fixing time for a bug of a particular component c by a particular developer d, can be accurately estimated by the average time d fixes bugs for component c. [#m0288cf2]
    • Concerns: Two bugs can belong to the same component, but one is of high difficulty and another is of low difficulty. Also, one bug could be of high importance (and thus get fixed ASAP), while another is of low importance. Would these variations affect bug fix time of individual bugs? Furthermore, some developers only fixed 1 bug from a particular component, while some others fixed 1000 bugs from a particular component. The estimate (using average fix time) for the first group of developers would be much poorer than the estimate for the second group of developers. [#r4eeb46e]
    • Assumption: Assigning most of the bugs to a few developers would make these developers overwhelmed. [#o5fd3802]
    • Concerns: The fact is most bug reports are fixed by a small group of "active developers"[19]. It is unclear why this is the case? Could it be possible that these active developers devote much of their time (voluntarily) to open source development while the other developers only devote little of their time to open source development? If this is the case, then assigning 10 bugs to developers that spend 10 hours a day would be lighter than assigning 1 bug to developers that spend 10 hours a month. [#nbed2981]
  5. The value of L affects the number of bugs that are successfully assigned and the time needed to resolve the bugs. To make the comparison fair, I think there is a need to set the value of L such that the number of bugs that are successfully assigned by the proposed approach is the same with those successfully assigned by SVM or real assignments.
  6. The constraints used for integer programming seems a bit wrong to me. In particular there is no constraint that specifies that a single bug can only be assigned to one developer.
  7. Effectively only bug reports from one system is analysed: Eclipse. There is a need to analyse more systems.
  8. Are the differences in the time needed to fix a bug significant? A statistical test can be used to answer this question.
  9. During assignment, does the proposed approach take into consideration whether a previously assigned bug report has been completed by X%?

Minor:

  1. Section 3.1 can be extended to describe integer programming better. The current description requires readers to understand integer programming to appreciate it.
  2. The description of time limitation (L) can be improved. In particular, I don't quite understand: "The value of P_{ij} depends on the time limitation L. The value of each P_{ij} can always be 5 if L is infinite". There is a need for further explanation.
  3. Many of the related works listed in the paper are rather old and newer papers on similar topics have been published.

トップ   新規 一覧 検索 最終更新   ヘルプ   最終更新のRSS