大平/査読結果/ICSME2014 の編集

[[大平/査読結果]]

*ICSME2014 Reviews [#ne2122c1]
Dear Masao,

We regret to inform you that we are unable to include your submission, Bug Triaging as An Optimization Problem: 0-1 Integer Programming for Mitigating Imbalanced Task Assignments in Large-Scale OSS Development, in the Research Track 
of ICSME 2014. In total, we received 267 abstracts and 210 full paper submissions. Only 40 of these 
papers were accepted, yielding an acceptance rate of 19%.

All papers went through a rigorous reviewing process in which they received at least three independent 
reviews, followed by discussions among the assigned reviewers, and, finally, a discussion open to the 
entire program committee. The program committee selected papers for inclusion in the program 
without targeting any quota or acceptance rate: merits and weaknesses of each paper were discussed 
independently of other papers taken into consideration.

Attached to this email, you will find the reviews of your paper. We hope that you will find them helpful in 
the revision of your paper for other ICSME tracks or other venues. In particular, we hope that you will 
consider submitting your work to some of the other tracks of ICSME 2014: the Early Research 
Achievement (ERA) Track, the Industry Track, the Tool Demo Track, and the Doctoral Symposium; 
especially if the reviewers suggest you to do so. Note that the deadline for abstract submissions for 
these tracks is June 23 (AoE), the deadline for full submissions (4 pages) is June 30 (with grace period til 
July 7). More information on these can be found at http://www.icsme.org/call-for-papers

You may also consider submitting your work to one of the workshops co-located with ICSME 2014. 
More information on those can be found at http://www.icsme.org/2014

We sincerely hope that you will attend ICSME 2014 in Victoria and look forward to seeing you there 
September 28 - October 3, 2014.

Thank you for submitting your paper to ICSME 2014

Best regards,
Leon Moonen and Lori Pollock
ICSME 2014 Program Co-Chairs

**----------------------- REVIEW 1 --------------------- [#y9f436b9]
This paper proposes task assignment optimization for OSS projects.Combining integer programming and
SVM machine learning method, the authors propose an assignment algorithm with workload limit for 
each developer. The proposed algorithm and traditional SVM approach have been investigated with
two projects, Firefox and Eclipse, and the proposed approach shows better performance of the
balanced assignment and total bug fix time.

＋ Overall, i enjoyed reading this paper. The paper proposes a new practical method of task allocation
for the OSS projects, where small set of developers tends to get many tasks.This approach clearly
sets the limitation of the total task load, and the solution would follow this constraint. Also,
it shows the reduction of the total bug-fix time with this new algorithm.

＋ An empirical validation has been performed with two large data sets, Foirefox and Eclipse, and 
the results clearly show the advantages of the proposed approach.

以下マイナスポイント
- It's not clear if the proposed approach would work interactively or not. I would guess that
the system gets the bug-fix allocation history and the new bug cost, and it answers the best-fit
allocation for that new bug. Or It would compute the optimal allocation for the whole bug data
of the target systems at the same time. In the latter case, practical use of this approach might be difficult.
Authors should clearly describe the proposed approach in such sense.Fig.1 and Fig.3 do not help
to understand it.

- Explanation of the model and formula is not sufficient. For example, formula (1)~(6) give
only limited overview of the model, so simple example of those formula would help to understand
more in depth.

- In the experiment, the authors have used a previous approach. However, it is not clear that
the previous approach would be constructed with sufficient consideration of balanced assignment.
I would guess that the older approach had different perspective and target, rather than balanced
assignment. Authors should clearly present this issue.

- Explanation of Fig.1 does not fit to the elements of Fig.1. Maybe more clear explanation would be
needed for Fig.1.

- Meaning of "common" is not clear.

- For Table VI, we would guess that the difference of SVM and IP is the advantage of this approach.
Is the difference significant? I'm not sure we would do such statistic in this case, but the authors
would mention on this issue.

- It seems that the approach assumes each bug is independent; no relations or no dependence. But such assumption is unrealistic I believe. It is natural such relations/dependence may raise/reduce the cost of each bug.

- What is the actual "objective functions?" It is clearly defined in Section 2,but it would be a tough work to define preferences and objective variables (and also define their values of each developer). Are there any examples of preferences/objectives?

- As time goes by, time limitation L might be changed (and also it depends on each developer).The approach employs L is constant integer; is it safe assumption for Firefox/Eclipse case?

- As the authors described in Section 7, it would be interesting to apply the approach to individuals, and it would be a chance to do a experiment with some subjects.

-------------------------------------------------------------
#comment()

**----------------------- REVIEW 2 --------------------- [#p92c1b42]
Summary:

In this paper, the authors formulate bug triaging as integer programming, an optimization problem. That is, given a set of developers and a set of bugs need fixing, bug triaging is to assign each task to a developer, such that the assignment maximizes the total "preference" of developers over bugs while keeping the total "fixing time" of each developer within his/her time limit. The "preference" is computed by a SVM-based method proposed in prior work and the "fixing time" is computed as the average fixing time of each developer on bugs of the same component previously fixed by him/her.

Comment:

Overall, the authors propose a novel approach for an important problem of optimal bug assignment, with multiple developers and multiple bugs. The paper is well-written and easy to understand. However, I have some concerns with the proposed problem and solution.

1. I wonder why the optimal bug triaging needs to be formulated as integer programming (IP) and why fixing time (C_ij) and time limit (L) needs to be integers? As far as I know, IP is NP-hard and 0-1 IP (as in this case) is NP-complete, thus, are very computational expensive. The authors report that "in our experimental setting, a calculation only took about two minutes for the two projects." This fast performance might be due to the small size of experiment data (e.g. just around 20 developers (Table V) and 40-80 bugs/month (Figure 5)). However, software projects might have thousands of developers and receive hundred bugs each day. In such case, IP would be exponentially more computational expensive and thus, is not practical.

2. Why the goal of bug triaging is to maximize the total "preference"? If we want to reduce fixing time and avoid over-assignment, why don't we try to minimize the total fixing time, or some cost/utility functions involving the allocated tasks (e.g. the unbalance of task assignments between developers). More seriously, the "preference" here is *predicted* by the SVM-based model, which has prediction errors and cannot be measured or quantitatively validated. Summing "preference" is also troublesome to me, as I cannot interpret the practical meaning of this "total preference". (The preference score can be used to rank developers for an individual bug, but I don't think we can add "preference" of two different bugs).

3. The proposed formulation does not take into account the time and semantic dependency between bugs. For example, a bug B needs to be fixed before a bug B' can be fixed, while B' might even be filed long before B. Thus, although B' is filed very soon, it should not be assigned right away.

4. The computation of fixing time of a developer on a bug is tricky. First, a developer might be assigned several bugs at the same time. He/she might be also assigned to several other tasks. Thus, it is difficult to measure the actual time he/she spends for a particular bug. The authors do not explain in details how they determine the fixing time in their experiment.

5. Predicting fixing time of a developer for a bug (C_ij) as "average time to fix bugs which are categorized into the same component" would be pretty inaccurate. As seen in Table III, fixing time varies significantly (e.g. from 1 day to 40 days). Thus, the average time (either mean or median) will be a poor estimate of the actual fixing time. This predicting method also makes the proposed approach less robust, as it cannot assign bugs to developers who have not fixed relevant bugs before.

6. The experiment setting has some flaws. For example, bugs should be sorted by filing/fixing time and bugs in training set should be filed/fixed before bugs in testing set.

-------------------------------------------------------------
#comment()

**----------------------- REVIEW 3 --------------------- [#v48c54f7]
This paper proposed a new capacity-aware bug triaging technique using the combination of 0-1 integer programing and SVM.  It assigns bug fixing tasks in consideration of a developer's workload capacity. The authors formulated the assignment task as a 0-1 integer programming problem.  The simulation-based evaluation was performed using two popular open source projects, and the results shows that days to fix were reduced and developers' loads were balanced.

This paper is well-structured and the formulation to a 0-1 IP is interesting. The evaluation is not complete because it is simulation-based and its data set is not so large; however, the careful analysis and discussion can successfully support the conclusions. Though more evaluations are needed and there are many future work issues, this study is promising.

To improve the paper, please consider the following points:

Since 0-1 IP problems are NP-hard, the scalability of the execution time of this technique should be discussed. The authors already wrote "two minute for the two project" in Sec.III; however, more general and deep discussion would help readers.

An example of Preference Pij and the intent of Exp.5 is not presented. Besides, the definition of Pij is absent in the paper. Since [2] does not describe how to calculate such score explicitly, its minimal explanation should be described here.

MINOR ISSUES:
- Figure 6, 7 and 8 are too small to read.
- Figure 2, 4, and 5 have no Y-axis labels.
- The explanation of "common" at Sec.IV-A is hard to understand and needs more words.
- The calculation of Cij is ambiguous and needs a more concrete description.
- In p.3, "our our" -> "our"
- At the 4th paragraph in Sec.IV-B, the reason should be described briefly, and an indentation is missing.
- In Table VI, "the less period" -> "the less busy period"
- The first paragraph in Sec.III-B 6) is unclear. L is not the parameter for Pij or the objective function but the parameter for this technique.
- At the 1st paragraph in Sec.V-A, "a tool large" -> "a too large"
- At the 1st paragraph in Sec.V-B, "is is" -> "is"
-----------------------
#comment()

タイムスタンプを変更しない

[[大平/査読結果]]

*ICSME2014 Reviews [#ne2122c1]
Dear Masao,

You may also consider submitting your work to one of the workshops co-located with ICSME 2014. 
More information on those can be found at http://www.icsme.org/2014

We sincerely hope that you will attend ICSME 2014 in Victoria and look forward to seeing you there 
September 28 - October 3, 2014.

Thank you for submitting your paper to ICSME 2014

Best regards,
Leon Moonen and Lori Pollock
ICSME 2014 Program Co-Chairs

＋ An empirical validation has been performed with two large data sets, Foirefox and Eclipse, and 
the results clearly show the advantages of the proposed approach.

- Explanation of Fig.1 does not fit to the elements of Fig.1. Maybe more clear explanation would be
needed for Fig.1.

- Meaning of "common" is not clear.

- As time goes by, time limitation L might be changed (and also it depends on each developer).The approach employs L is constant integer; is it safe assumption for Firefox/Eclipse case?

- As the authors described in Section 7, it would be interesting to apply the approach to individuals, and it would be a chance to do a experiment with some subjects.

-------------------------------------------------------------
#comment()

**----------------------- REVIEW 2 --------------------- [#p92c1b42]
Summary:

Comment:

6. The experiment setting has some flaws. For example, bugs should be sorted by filing/fixing time and bugs in training set should be filed/fixed before bugs in testing set.

-------------------------------------------------------------
#comment()

To improve the paper, please consider the following points:

テキスト整形のルールを表示する