reviews/wcre2011 の編集

[[reviews]]

* 18th Working Conference on Reverse Engineering (WCRE 2011) [#d6ae2e5e]

** WCRE 2011 notification for paper 107 [#h89a14db]
Dear Masao,

Thank you for your submission to the 18th Working Conference on Reverse Engineering (WCRE 2011) to be held in Limerick, Ireland, October 17-20, 2011.

We are pleased to inform you that your paper:

The Impact of Bug Management Patterns on Bug Fixing: A Case Study of  Eclipse Projects

has been accepted as a SHORT paper for presentation at WCRE 2011 and publication in the conference proceedings.

Papers went through a rigorous reviewing process. Of 104 papers submitted, 27 were accepted as full papers and 23 more as short papers.

We enclose below the reviewers'  comments to your submission. You are expected to address the suggestions of the reviewers carefully, when
preparing the final, camera-ready copy of the paper. SHORT papers are limited to 5 pages following the IEEE standards.
(please check http://www.computer.org/portal/web/cscps/formatting).

If the paper has been submitted as FULL paper, please let us know whether you are accepting our offer for the SHORT paper until August 11th 2011 by
sending an email to the Program Co-Chairs.

Your camera-ready paper is due on August 20, 2011. The information about the IEEE online author kit will be sent to you in a separate email the
next couple of days.

It is a requirement that at least one of the authors registers and presents the paper during the conference.

Please also consider submitting to the Doctoral Symposium. The deadline is August 9th, the investment is small, but the payoff is big: the PhD student
will receive feedback on his/her PhD studies and other tips & tricks from two community experts, Radu Marinescu and Michele Lanza.

Congratulations on having your paper accepted. We look forward to seeing you in Limerick.

Sincerely,

Denys Poshyvanyk and Martin Pinzger
WCRE 2011 Program Co-Chairs

**----------------------- REVIEW 1 --------------------- [#vaa2a095]
PAPER: 107
TITLE: The Impact of Bug Management Patterns on Bug Fixing: A Case Study of  Eclipse Projects
AUTHORS: Masao Ohira, Naoya Osawa, Ahmed E. Hassan and Ken-Ichi Matsumoto

Summary
------------

The authors present an empirical study carried out to evaluate the impact of bug management on bug
fixing. In the context of the study the authors analyse how the communication between different actors,
namely bug reporter, triager, and fixer, impact on the time needed to assign the bug and the time needed
to fix it. In particular, the authors analyse four different patterns that distil the different relation between
the actors involved in the bug management process. The results achieved indicate a significant impact of
these patterns on the efficiency of the bug management process.

Comments
------------

This is a very interesting paper. It is well written and easy to follow. The study is well motivated.
However, a deeper analysis of the achieved results is needed to provide actual insights to improve the
bug management process. To this aim, a deeper qualitative analysis of the results is required, in
order to take into account other factors that could affect the bug management process, e.g. bug
severity.

In addition, I did not completely understand the rationale behind the explorative study presented in this
paper. The explorative study seems to be useful only (i) to identify the patterns that distil the different relations
between the actors involved in the bug management process and (ii) to have a preliminary indication that
the hypotheses made by the authors make sense. I really like explorative study and I think that they are
extremely useful.

However, in my opinion it is not necessary, for the economy of this paper, to describe it in such a detail.
Indeed, the authors could avoid the description of the explorative study and provide more qualitative analysis
on the case study on bug management process. At the end of the day, the conjectures made by the authors
are reasonable and probably they could be derived also without an explorative study. So I strongly suggest to
remove Section III (note that all the findings are also included in Section V) and only mention the explorative
study in Secion IV saying that the patterns were derived from an explorative study performed on Eclipse.

In this way the authors can save almost on page. They can use this space to improve the discussion of two
important issues related to the case study:
1. how does the quality of the bug report interact with the patterns identified by the authors? How does the
bug management process is affected by the skills of the bug reporter (even when it is not a customer)?
2. provide more details on how the communication between the different actors impacts the bug management
report. Do the authors have any information about the number of emails exchanged by the different actors?

Summarising, the main merit of this paper is that it provides the first attempt at describing a phenomena (bug
management process) taking into account not only information derived from bug reports and source code
files, but also putting humans in the loop considering the interaction between people involved in that process.
However, a deeper analysis of the results is required to make this study really useful.

**----------------------- REVIEW 2 --------------------- [#p82d03ad]
PAPER: 107
TITLE: The Impact of Bug Management Patterns on Bug Fixing: A Case Study of  Eclipse Projects
AUTHORS: Masao Ohira, Naoya Osawa, Ahmed E. Hassan and Ken-Ichi Matsumoto

Summary
------------

The paper presents an empirical study, performed on Eclipse Platform and JDT, that investigates the relationships
between certain bug management process patterns and the time to assign and fix bugs. In particular, the authors define
4 patterns: 1) the reporter, the triager and the fixer are the same person, 2) they are all different people, 3) the reporter
and the triager are the same person, while the fixer is another one, and 4) the triager and the fixer are the same person,
while the reporter is a different one.
In the study, the authors considered only bugs from 2007 to 2009, which were not reassigned (without tossing), ending
up with ~4000 bugs for the Eclipse platform, and ~1600 for JDT.
The study revealed that: when a reporter is also the triager, the time to assign the bug is shorter; when a triager assigns
a bug to fix to himself, such assignment takes longer than when he assigns bugs to other developers, but at the same
time, the bug fixing is up to twice faster; when the reporter, triager and fixer are different people, the total time required
is longer than in all other cases.
The authors argue that these results imply that a better way to share knowledge between a project's personnel would
improve the bug management process, and thus decrease the time to resolve bugs.

Evaluation
--------------

The study presented in the paper is novel, interesting and well motivated. In particular, I appreciated that the authors
extract the motivations for performing such a study from a survey of the literature. Further, starting the analysis and the
paper with a pilot study improves the readability of the paper on the one hand, and provides additional motivations on
the other hand.
Having said that, I have one major problem with the paper: the authors do not consider a number of factors which impact
the time for assigning and fixing bugs, and just assume that the reason for which some bugs take longer to fix (and/or
assign) than others is lack of communication. For example, severity and priority of bugs are crucial factors which highly
impact the fixing time: they are neither considered nor mentioned by the authors. According to the authors, the fact that
bugs with triager different from fixer take longer to fix is due to poor knowledge sharing. What I argue is that this is only
one possible reason. For example, let us consider bugs with low severity and priority (not important and not urgent to
fix); according to the authors, triagers are senior developers, and it is quite unlikely that senior developers focus on
unimportant bugs, so it is quite likely that the triager is different from the fixer. In such a scenario, the fact that bugs
take long to fix is perhaps due to their low priority: since they are not important, they will be put at the end of the "to-
do" list by developers.
Because of the above reasons, in their analysis the authors should consider, at least, priority and severity of bugs, or
provide convincing motivations of not doing so.
In short, I think that while the analysis that the authors performed is interesting, the interpretation of the results is over
simplified, thus leading to claims that are too strong with respect to the actual facts supported by the data. The authors
should elaborate more their interpretation and tune the claims down.

Moreover, I have the following remarks about the paper:

-The research questions RQ1 and RQ2 are too general and, in my opinion, cannot be answered with the data used in the
study. For example, RQ1 is "Does the type of reporter-triager relationship affect the time to assign a bug fixing task?".
However, the data used in the experiment is not sufficient to characterize the reporter-triager relationship: it just
distinguishes between the case in which the reporter and the triager are the same person, and the case in which they are
different people. This can be hardly defined the "reporter-triager" relationship. RQ1 should be rephrased in something
like: "Does the fact that who reports a bug and who triages it are the same person affect the time to assign a bug fixing
task?"
The final finding of the section (reported in the rounded box) should also be rephrased accordingly. The same holds for
RQ2.

-For the study, the authors filtered out bugs which were reassigned (with tossing). While I agree with this choice, it has
the drawback of selecting only part of the bug population, which might create a bias, thus limiting the generalizability of
the results (see for example "Fair and balanced?: bias in bug-fix datasets" http://portal.acm.org/citation.cfm?
id=1595716). Moreover, the authors only considered the bugs reported from 2007 to 2009... why? Expanding the
considered time window would create a much larger dataset, thus making the results stronger. The authors should
discuss the potential bias in the threats to validity.

-At page 7, the authors state "This phenomena likely happened when a triager in [R<>T=F] tried to ask a fixer to take a
bug fixing task but cannot get any response from the fixer (or cannot find an adequate fixer) and finally assigns the task
to himself". Did the authors verified this fact on the data? Which confidence do they have with such a conclusion?

-Page 7, "We manually checked 100 randomly selected bug reports of the whole bug report data". Why did the authors
select 100 samples and not 50 or 150? Does this number provide some sort of statistical significance?

-I was surprised of not finding an in depth analysis of the total time to fix a bug (i.e., Tassignment + Tfix), as in the end
this is what really matters. The authors somehow discuss the combination of Tfix and Tassignment, although quite
superficially, but in my opinion what is needed is to study the total time using it as a variable, and running the statistical
tests on it.

-Page 8, "Figure 8 shows the relationship between the bug-fixing time and the number of discussions comments before
the bug assignment in the Platform project. We see Tfix is gradually reduced as comments (discussions about a bug in
Bugzilla) increase". This is really interesting but a bit too shallow. Expanding this analysis would be beneficial for the
paper. The authors can find the necessary space by shortening the description of the patterns, which is too verbose.

-At page 3, the authors state "If an individual plays both the roles of reporter and triager, we considered him/her as a
developer. If not (reporter &#824;= triager), it is likely that a reporter is a user". Why if the reporter is not the triager, then the
reporter is a user? As the authors themselves state at page 2 "A triager is often a senior developer who has the authority
to assign a bug fixing task to other developers in the project". Therefore, if the reporter and the triager are two different
people, they can be for example a senior developer (triager) and a junior one (reporter).

Concerning the presentation, the paper is fairly well structured, but I found a number of grammatical errors and typos,
listed in the minor comments below.

All in all, what the authors should do, in my opinion, is:
- tune down their claims
- reformulate the research questions
- acknowledge the fact that tassign and tfix are influenced by a number of factors, some of which are easy to filter as
they are structured information (e.g., severity, priority)
- address the other remarks discussed in the review

Points in favor and against:
+ good motivations
+ interesting study
+ conservative statistical tests used (Mann-Whitney)
- the interpretation of the results it too simplified, without considering many factors which might affect the fixing time
- the claims are too strong with respect to the support given by the data
- some research questions are too general and cannot be answered with the data at hand; they need reformulation

Minor comments
----------------------

--Figures and tables should be moved closer to the text that describes them.

--The description of the Bug Management Patterns is too verbose and can be shortened.

--page 1, "is the first attempt at empirically study" -> studying

--page 1, "different individual playing different roles" -> individuals

--reference [12] is now a TSE publication. See: http://www.computer.org/portal/web/csdl/doi/10.1109/TSE.2010.63

--at page 2, the authors introduce the abbreviation BTS (bug tracking system). This should be done the first time they
mention it, i.e., at page 1 "Software projects use a bug tracking system"

--page 2, "since the goal of existing studies as previously described is to reduce" -> existing studies, as previously
described, is to reduce

--page 2, "The triager can also be a reporter and/or a fixer as." -> kill "as"

--page 2, "In such the case, the efficiency of the bug management process should be different from the case where the
each role" -> In such a case, the efficiency of the bug management process should be different from the case where each
role

--figure 1, figure 2, and page 7, triagger -> triager

--"Table II shows the results of our analysis on Tfix. We found that there were significant differences in Tassignment for
both projects." -> Tfix for both projects

--page 4, "The are defined by combinations" -> They are

--page 4, "his ability to fix the fixing bug." -> fix the bug

--page 5, "We can see that about half of reported bugs were fixed without tossing (reassignment)". This is wrong.
Looking at table III, the total number of fixed bugs is 8'434 + 4'133 = 12'567. The bugs which were fixed without
tossing are 4'133, so they are about 1/3 of the reported bugs, or about half of the bugs with tossing.

--In figure 5 and 7, using different line styles, instead of different colors, would make the figures more readable.
Moreover, the figures should be zoomed on the y axis, as parts of them (0-0.6 in figure 5, and 0-0.3) are useless.

--page 7, "We can see that the differences at 1% significant level between [R≠ T≠ F] and other patterns in Platform and
JDT". -> A verb is missing in this sentence.

--page 7, "to assing the tasks" -> assign

--page 9, "Some projects (i.e., JDT) put careful attention on prioritizing the fixing of bugs" -> It is not "some projects",
but *one* project.

**----------------------- REVIEW 3 --------------------- [#tc965340]
PAPER: 107
TITLE: The Impact of Bug Management Patterns on Bug Fixing: A Case Study of  Eclipse Projects
AUTHORS: Masao Ohira, Naoya Osawa, Ahmed E. Hassan and Ken-Ichi Matsumoto

Summary
------------
This paper investigates the social aspect of the bug management process and how this has an effect on the actual time needed to fix a bug. The social aspect is classified as "bug management patterns" = the various possible combinations between the reporter, triager and the actual fixer of the bug. Based on observations from tow cases, the authors concluded that as the number of actors involved in the process increases, so will the time increase to fix a bug. Furthermore, it is stated that when an actor assigns a bug to himself for fixing, the global time needed to fix a bug increases.

Points in favour
------------
-The paper is well written
-studying the social aspect of bug-triaging and classifying them as bug management patterns is an interesting idea

Points against
------------
-when I read "bug management patterns" I was expecting something similar as "design patterns", i.e. a qualitative analysis of good practices that have proven their worth in at least 3 different situations and who receive a meaningful name. I was quite disappointed with what the authors called patterns in their paper. First of all, it's based on quantitative analysis of the occurrences of the three possible combinations of reporter eq/neq trigger eq/neq fixer -- pretty uninteresting if you ask me as it is in only few cases that one of them equals another. Moreover, the authors claim that these combinations have a causal relationship on the bug fixing time, this way suggesting that bug triggers should take them into account. For instance: the authors suggest that a trigger assigning a bug to a fixer and afterwards (due to a lack of response) reassigns it to himself takes a long time -- how can one take that into account; the root cause is out of control.

--case study is too small to provide results that can be generalized: only two cases from a single open source project is used in this study. More cases from across different projects should be included in the study.

--Detailed Remarks (optional)
--- include page numbers when submitting a paper for review, it's easier to refer to spelling mistakes
--- last sentence of first paragraph: "bug reports from on a daily basis" -- some word is missing here.
--- triagging -> triaging
--- the total number of reports as stated in Section 6 (at the bottom of the page) does not match with the numbers in Table 1 and 2.

タイムスタンプを変更しない