大平/査読結果/SSE2013
の編集
https://floss-lab.org/pukiwiki/?%E5%A4%A7%E5%B9%B3/%E6%9F%BB%E8%AA%AD%E7%B5%90%E6%9E%9C/SSE2013
[
トップ
] [
編集
|
差分
|
履歴
|
添付
|
リロード
] [
新規
|
一覧
|
検索
|
最終更新
|
ヘルプ
]
-- 雛形とするページ --
(no template pages)
[[大平/査読結果]] *SSE 2013 notification for paper 8 [#a9b9940d] Dear Masao, Thank you for your submission to SSE'13 workshop at ESEC/FSE'13. We are pleased to announce that your paper: A new perspective on the socialness in bug triaging: A case study of the Eclipse Platform project was accepted as a SHORT PAPER for our workshop. Congratulations! Each submitted paper was reviewed by at least three members of the Program Committee. The overall quality was high, and the Program Committee had a difficult time selecting from among the many high-caliber papers. Please find your reviews attached. Please revise your paper according to the reviewers' comments. Make sure that you follow the ACM SIG Proceedings formatting guidelines. - You cannot exceed 4 pages for your paper. - The deadline is July 1, 2013 - You will be contacted by the publisher soon with an author-kit email. Please note: At least one author of your paper must be registered for the workshop. A detailed workshop agenda will be available within the next weeks at the workshop homepage www.socialse.org. We will inform you in a separate mail about the agenda and how you should prepare to present your work. Best regards, The SSE’13 Organizers **REVIEW 1 [#lccad244] OVERALL EVALUATION: 1 (weak accept) REVIEWER'S CONFIDENCE: 5 (expert) ***REVIEW [#f405ebc4] The paper presents a new perspective to bug triaging by stressing the question of ‘who should assign this bug’ in contrast to the classical approach of ‘who should fix this bug’. The topic discussed is interesting and relevant to the workshop. Overall the paper is well motivated and nicely written. However, the paper could be improved with respect to the following points: - The authors need to explicitly define “social relationships” and “fixing performance” listed in the research questions. The two terms could have different meanings in different contexts. - The authors need to include a new section on “Study design” before presenting the case study. The new section should be structured as follows: case and subject selection, data sources, data collection procedure, and analysis procedure. - The authors need to discuss whether the new perspective could be associated in some cases with performance overhead or other limitations. Could the new perspective “disturb” the way open source communities work! - The authors need to discuss concrete practices and tools in order to implement their new perspective. **REVIEW 2 [#q494c77a] OVERALL EVALUATION: -1 (weak reject) REVIEWER'S CONFIDENCE: 4 (high) ***REVIEW [#n6bb947b] This paper tries to add a new perspective to bug triaging by investigating social relationships among the persons who assign bugs and the persons who actually fix the bugs. The authors want to find out, if such a social relationship influences (1) the time and (2) the "performance" of the bug fixing task. The paper is overall quite well written as far as language is concerned, but it includes spelling errors and several sentences appear multiple times in the document. It could probably be shortened by nearly one page and still be understandable. However, my biggest concerns are fundamental doubts regarding the research setting: 1. I cannot agree with what the authors call "social relationship" between assignors and fixers and how they try to measure it (construct validity): What is the authors' evidence that the number of bugs which were assigned by person A to person B affects their social relationship in any way? I am not sure if this is a problem of the notion "social relationship", but it does not seem right. We do not have evidence that A knows B and even less evidence that B knows A. This point invalidates the study. 2. As answers to their RQs, the authors make two statements, which they try to back up with the independence or dependence of two variables. But in both cases the sample size is too small and the method (manually comparing the medians) is not sufficient. In neither case do the authors use a statistical test. 3. The authors do not explain from where they get the time to fix a bug. Was that from the time of assignment until the time of closing or resolving? Or did the fixer track her progress? This makes a big difference and therefore is an important fact to name. The huge deviation among the median days to fix bugs might also be a result of this value being subjective. 3a. It might be that a specific assignor is typically concerned with bugs that require more effort (say UI bugs?) or with a part of the system which has more dependencies to other subsystems, and therefore needs more development, testing, documentation, reading, communication and so on and so forth. While the question 'does the bug fixing performance of a fixer depend on who assigns tasks to the fixer' might even be answered, I fear there is not enough evidence from comparing such a small number of medians. At least, if one really wants to approach the question in this way, statistical independence has to be tested with an appropriate test over a large sample. But in particular in the context of the paper, which claims to clarify if "social relationship" could be the cause or not the cause for such a phenomenon, this does not seem to be the right method. 3b. Why do the authors use medians, why not means? It might be right, but the authors need to tell us something about the distribution to justify that. Is it positively skewed? 4. The preparation steps used to filter the bugs are unclear to me. 5. The external validity of the study is not given, as only a single case was studied. This might even be ok, if (a) the title of the paper includes "OSS", (b) the threats to validity section does not claim so, and (c) the statements are transformed into hypotheses. Nevertheless, the authors claim that the size of Eclipse is sufficient "to validate results" of the case study. I disagree with that. But as an advice, I would simply not claim external validity for a single-case case study not even with backup. There are also other points which the authors should consider or improve: -What do the authors think about bug priority, which is typically used to drive fixers' efforts? Is it really relevant to discuss about the work time of the fixer from the assignor's perspective? And if yes, is this not even more relevant in industrial settings than open source? -How do these studies relate to hierarchies or groups (subcommunities) among fixers? At the time of re-assignment, the "wrong" fixer becomes the assignor. Is this covered by the current study? In particular, fixers might now each other even better than assignors know fixers, or also not - we simply don't know that. So the study cannot just exclude that, if it claims to analyze "social" aspects. -On an individual level, what is the difference between the set of assignors and the set of fixers? Are they disjoint? Was that studied? I think it is an important question, since the study aims to uncover the social aspects. -How does the social structure between developers in open source projects relate to the one in industrial companies? Are the research questions relevant in such scenarios? -The discussion section 4.1 is unclear to me. -The paper includes several unsubstantiated claims, such as that preventing bug tossing is very effective in reducing the time needed to fix bugs, or that social network analysis is not an adequate tool to analyze fine-grained social aspects in software development. Some of them might be backed up by literature, others at least not to my knowledge. If the authors stick with this study, I would really recommend new clothes for it: It is a case study. If the results, which should be called hypotheses in any case, are interesting, the authors do not need to claim their generalizability. Instead, the authors should simply report on the particular interesting observation in a single case and make it clear that additional studies are necessary for this single observation to become an established fact. In any case, the authors should think about the term "social relationship" and what it really means. It could be that a discussion of that notion, and narrowing it down to a more specific property could help. **REVIEW3 [#t5d340d4] OVERALL EVALUATION: 1 (weak accept) REVIEWER'S CONFIDENCE: 4 (high)***REVIEW ***REVIEW [#w56c2075] The authors ask some interesting questions about the impact of who assigns bugs on the efficiency of bug fixing in an open source project (Eclipse Platform). They provide a nice review of the related work and emphasize that most of the related work is on who should fix a bug rather than considering who should assign a bug. The case study (Eclipse platform) is appropriate for an exploratory look at this question. The finding from RQ2 is interesting and could lead to an interesting discussion at the workshop. Some areas for improvement are as follows: - The motivation was not so compelling to me at first because I questioned if it is possible to really change who assigns bugs -- can you elaborate how that could be influenced? - RQ1 was not clearly defined especially in terms of how "social relationships" are conceptualized. By social relationship I think you just mean familiarity? But there are other kinds of social relationships. - Surely some fixers (and assigners) focus on certain parts of the Eclipse platform code -- wouldn't that play a role? Did you consider this? - Why not use consistent labelling for Fixers and Assignors? (sometimes they are F1 etc versus Fa). - Section 4.1 doesn't really discuss productivity vs. efficiency -- it simply puts the results from the tables in words. But some discussion here would be great! - I really think you should consider the contents and properties of bug reports -- couldn't this be done fairly quickly especially the properties? - Why just consider top five fixers? Can't the analysis be automated to look at more? This may give a skewed picture perhaps? - What about the impact of managers from IBM (or power over relationships) on time to fix bugs? This seems a more important social relationship than familiarity. Writing issues: - The paper is quite repetitive in places - The English needs improvement, lots of typos and English issues. But the paper is nicely structured. **REVIEW4 [#ze6281fc] OVERALL EVALUATION: -1 (weak reject) REVIEWER'S CONFIDENCE: 4 (high) ***REVIEW [#u667ea1d] The idea behind this paper is intriguing and I wonder why you started to ask the question? Is it because you found some comment regarding a fixers perspective on who gave them the bug to fix? You have background on bug triaging in the paper, but dont explain why this is an interesting question to answer. The three main findings reported at the end of the introduction are a little confusing to me as they use different language (e.g. time to fix versus performance) which may or may not be significant, and Im not sure how to interpret them. So I suggest these things are explained in the intro; as I read the rest of the paper it becomes clearer what youre doing, but the intro does not need to be so confusing. I also suggest that you define what you mean by social relationship in the intro (it seems to mean that they have worked together often). Dont you have to take into account the type of bug in order to make these judgements? I see that you acknowledge this in your threats section, but you dont mention complexity. In addition, how have you arrives at your conclusions? You present descriptive statistics but dont look at any significance data. With quantitative data like this you need to apply more sophisticated statistical tests. This would make your results much stringer and allow you to use more precise language in making your conclusions. The paper would benefit from some checking regarding English language.
タイムスタンプを変更しない
[[大平/査読結果]] *SSE 2013 notification for paper 8 [#a9b9940d] Dear Masao, Thank you for your submission to SSE'13 workshop at ESEC/FSE'13. We are pleased to announce that your paper: A new perspective on the socialness in bug triaging: A case study of the Eclipse Platform project was accepted as a SHORT PAPER for our workshop. Congratulations! Each submitted paper was reviewed by at least three members of the Program Committee. The overall quality was high, and the Program Committee had a difficult time selecting from among the many high-caliber papers. Please find your reviews attached. Please revise your paper according to the reviewers' comments. Make sure that you follow the ACM SIG Proceedings formatting guidelines. - You cannot exceed 4 pages for your paper. - The deadline is July 1, 2013 - You will be contacted by the publisher soon with an author-kit email. Please note: At least one author of your paper must be registered for the workshop. A detailed workshop agenda will be available within the next weeks at the workshop homepage www.socialse.org. We will inform you in a separate mail about the agenda and how you should prepare to present your work. Best regards, The SSE’13 Organizers **REVIEW 1 [#lccad244] OVERALL EVALUATION: 1 (weak accept) REVIEWER'S CONFIDENCE: 5 (expert) ***REVIEW [#f405ebc4] The paper presents a new perspective to bug triaging by stressing the question of ‘who should assign this bug’ in contrast to the classical approach of ‘who should fix this bug’. The topic discussed is interesting and relevant to the workshop. Overall the paper is well motivated and nicely written. However, the paper could be improved with respect to the following points: - The authors need to explicitly define “social relationships” and “fixing performance” listed in the research questions. The two terms could have different meanings in different contexts. - The authors need to include a new section on “Study design” before presenting the case study. The new section should be structured as follows: case and subject selection, data sources, data collection procedure, and analysis procedure. - The authors need to discuss whether the new perspective could be associated in some cases with performance overhead or other limitations. Could the new perspective “disturb” the way open source communities work! - The authors need to discuss concrete practices and tools in order to implement their new perspective. **REVIEW 2 [#q494c77a] OVERALL EVALUATION: -1 (weak reject) REVIEWER'S CONFIDENCE: 4 (high) ***REVIEW [#n6bb947b] This paper tries to add a new perspective to bug triaging by investigating social relationships among the persons who assign bugs and the persons who actually fix the bugs. The authors want to find out, if such a social relationship influences (1) the time and (2) the "performance" of the bug fixing task. The paper is overall quite well written as far as language is concerned, but it includes spelling errors and several sentences appear multiple times in the document. It could probably be shortened by nearly one page and still be understandable. However, my biggest concerns are fundamental doubts regarding the research setting: 1. I cannot agree with what the authors call "social relationship" between assignors and fixers and how they try to measure it (construct validity): What is the authors' evidence that the number of bugs which were assigned by person A to person B affects their social relationship in any way? I am not sure if this is a problem of the notion "social relationship", but it does not seem right. We do not have evidence that A knows B and even less evidence that B knows A. This point invalidates the study. 2. As answers to their RQs, the authors make two statements, which they try to back up with the independence or dependence of two variables. But in both cases the sample size is too small and the method (manually comparing the medians) is not sufficient. In neither case do the authors use a statistical test. 3. The authors do not explain from where they get the time to fix a bug. Was that from the time of assignment until the time of closing or resolving? Or did the fixer track her progress? This makes a big difference and therefore is an important fact to name. The huge deviation among the median days to fix bugs might also be a result of this value being subjective. 3a. It might be that a specific assignor is typically concerned with bugs that require more effort (say UI bugs?) or with a part of the system which has more dependencies to other subsystems, and therefore needs more development, testing, documentation, reading, communication and so on and so forth. While the question 'does the bug fixing performance of a fixer depend on who assigns tasks to the fixer' might even be answered, I fear there is not enough evidence from comparing such a small number of medians. At least, if one really wants to approach the question in this way, statistical independence has to be tested with an appropriate test over a large sample. But in particular in the context of the paper, which claims to clarify if "social relationship" could be the cause or not the cause for such a phenomenon, this does not seem to be the right method. 3b. Why do the authors use medians, why not means? It might be right, but the authors need to tell us something about the distribution to justify that. Is it positively skewed? 4. The preparation steps used to filter the bugs are unclear to me. 5. The external validity of the study is not given, as only a single case was studied. This might even be ok, if (a) the title of the paper includes "OSS", (b) the threats to validity section does not claim so, and (c) the statements are transformed into hypotheses. Nevertheless, the authors claim that the size of Eclipse is sufficient "to validate results" of the case study. I disagree with that. But as an advice, I would simply not claim external validity for a single-case case study not even with backup. There are also other points which the authors should consider or improve: -What do the authors think about bug priority, which is typically used to drive fixers' efforts? Is it really relevant to discuss about the work time of the fixer from the assignor's perspective? And if yes, is this not even more relevant in industrial settings than open source? -How do these studies relate to hierarchies or groups (subcommunities) among fixers? At the time of re-assignment, the "wrong" fixer becomes the assignor. Is this covered by the current study? In particular, fixers might now each other even better than assignors know fixers, or also not - we simply don't know that. So the study cannot just exclude that, if it claims to analyze "social" aspects. -On an individual level, what is the difference between the set of assignors and the set of fixers? Are they disjoint? Was that studied? I think it is an important question, since the study aims to uncover the social aspects. -How does the social structure between developers in open source projects relate to the one in industrial companies? Are the research questions relevant in such scenarios? -The discussion section 4.1 is unclear to me. -The paper includes several unsubstantiated claims, such as that preventing bug tossing is very effective in reducing the time needed to fix bugs, or that social network analysis is not an adequate tool to analyze fine-grained social aspects in software development. Some of them might be backed up by literature, others at least not to my knowledge. If the authors stick with this study, I would really recommend new clothes for it: It is a case study. If the results, which should be called hypotheses in any case, are interesting, the authors do not need to claim their generalizability. Instead, the authors should simply report on the particular interesting observation in a single case and make it clear that additional studies are necessary for this single observation to become an established fact. In any case, the authors should think about the term "social relationship" and what it really means. It could be that a discussion of that notion, and narrowing it down to a more specific property could help. **REVIEW3 [#t5d340d4] OVERALL EVALUATION: 1 (weak accept) REVIEWER'S CONFIDENCE: 4 (high)***REVIEW ***REVIEW [#w56c2075] The authors ask some interesting questions about the impact of who assigns bugs on the efficiency of bug fixing in an open source project (Eclipse Platform). They provide a nice review of the related work and emphasize that most of the related work is on who should fix a bug rather than considering who should assign a bug. The case study (Eclipse platform) is appropriate for an exploratory look at this question. The finding from RQ2 is interesting and could lead to an interesting discussion at the workshop. Some areas for improvement are as follows: - The motivation was not so compelling to me at first because I questioned if it is possible to really change who assigns bugs -- can you elaborate how that could be influenced? - RQ1 was not clearly defined especially in terms of how "social relationships" are conceptualized. By social relationship I think you just mean familiarity? But there are other kinds of social relationships. - Surely some fixers (and assigners) focus on certain parts of the Eclipse platform code -- wouldn't that play a role? Did you consider this? - Why not use consistent labelling for Fixers and Assignors? (sometimes they are F1 etc versus Fa). - Section 4.1 doesn't really discuss productivity vs. efficiency -- it simply puts the results from the tables in words. But some discussion here would be great! - I really think you should consider the contents and properties of bug reports -- couldn't this be done fairly quickly especially the properties? - Why just consider top five fixers? Can't the analysis be automated to look at more? This may give a skewed picture perhaps? - What about the impact of managers from IBM (or power over relationships) on time to fix bugs? This seems a more important social relationship than familiarity. Writing issues: - The paper is quite repetitive in places - The English needs improvement, lots of typos and English issues. But the paper is nicely structured. **REVIEW4 [#ze6281fc] OVERALL EVALUATION: -1 (weak reject) REVIEWER'S CONFIDENCE: 4 (high) ***REVIEW [#u667ea1d] The idea behind this paper is intriguing and I wonder why you started to ask the question? Is it because you found some comment regarding a fixers perspective on who gave them the bug to fix? You have background on bug triaging in the paper, but dont explain why this is an interesting question to answer. The three main findings reported at the end of the introduction are a little confusing to me as they use different language (e.g. time to fix versus performance) which may or may not be significant, and Im not sure how to interpret them. So I suggest these things are explained in the intro; as I read the rest of the paper it becomes clearer what youre doing, but the intro does not need to be so confusing. I also suggest that you define what you mean by social relationship in the intro (it seems to mean that they have worked together often). Dont you have to take into account the type of bug in order to make these judgements? I see that you acknowledge this in your threats section, but you dont mention complexity. In addition, how have you arrives at your conclusions? You present descriptive statistics but dont look at any significance data. With quantitative data like this you need to apply more sophisticated statistical tests. This would make your results much stringer and allow you to use more precise language in making your conclusions. The paper would benefit from some checking regarding English language.
テキスト整形のルールを表示する
![[PukiWiki]](image/wadanyan.png "[PukiWiki]")
