ヤマタニ/IWESEP2014 の履歴(No.1)

履歴一覧
差分を表示
現在との差分を表示
ソースを表示
ヤマタニ/IWESEP2014 へ行く。
- 1 (2014-09-02 (火) 15:09:20)
- 2 (2014-09-03 (水) 21:50:34)

ヤマタニ

IWESEP2014＠中之島†

↑

title†

An Exploratory Analysis for Studying Software Evolution: Time-Delayed Correlation Analysis

↑

abstruct†

↑

review†

PAPER: 11 TITLE: An Exploratory Analysis for Studying Software Evolution: Time-Delayed Correlation Analysis AUTHORS: Yosuke Yamatani and Masao Ohira

OVERALL EVALUATION: 1 (weak accept) REVIEWER'S CONFIDENCE: 2 (low)

This paper proposes a method for an exploratory analysis of software evolution, called delayed correlation analysis (DCA). DCA extends the correlation analysis to consider a time gap between any pair of metrics, and returns the highest correlation coefficient for each pair of metrics. With various metrics, the authors conducted a case study and reported some detailed results.

Pros:

The problem of the correlation analysis of two time series with gaps should be interesting. I found some Q&A forms and papers. It should be more beneficial if the paper clarifies the differences with such previous approaches.

 http://stats.stackexchange.com/questions/7727/how-to-correlate-two-time-series-with-gaps-and-different-time-bases
 http://stats.stackexchange.com/questions/23993/how-to-correlate-two-time-series-with-possible-time-differences
 http://www.eecs.qmul.ac.uk/~ccloy/project_time_delayed/index.html
 http://maxi.riken.jp/FirstYear/proceedings/pdf/Scargle_Jeffrey_D.-bin-20110413063416.pdf
 http://maxi.riken.jp/FirstYear/ppt/O26SCARGLE.pdf
 http://www.itc.nl/~rossiter/teach/R/R_ts.pdf

The paper studied various metrics from three OSS projects with long histories.

Cons:

The result section is not easy to understand. For example, the statements (Section V-A, 2nd paragraph, 4th line and 3rd paragraph, 1st line) are not clear. In addition, I do not understand why the metric pair of # of bugs that severity are blocker and # of commentators is not selected in top correlations.
The proposed method tends to show illusory correlation more compared to the traditional correlation analysis. For an exploratory analysis, this is fine if the users of this method analyze the results carefully. Discussing this issue should be required.

Minor comments:

Section I, 3rd paragraph: in step after the delay from the programing development process. in step after the delay from the programing development process. -> in step after the delay from the programing development process.
Section III-B, 3rd paragraph: into a single data point. a value of -> into a single data point. A value of
Section IV-A, last line: grayed part in Figure 2 <- there is no grayed part, C^DCA -> CA^DCA

PAPER: 11 TITLE: An Exploratory Analysis for Studying Software Evolution: Time-Delayed Correlation Analysis AUTHORS: Yosuke Yamatani and Masao Ohira

OVERALL EVALUATION: 1 (weak accept) REVIEWER'S CONFIDENCE: 3 (medium)

Summary: This paper proposes a technique to take into account the existence of time lags in correlation analysis of software metrics. The authors conducted an empirical study with three OSS projects and 100 metrics. The authors checked all the possible combinations of the metrics with various setting values, and found some hidden correlations that cannot be found with traditional approaches.

Pros:

The considered problem is important.
The approach proposed in this paper makes sense to cope with the considered problem.
The empirical study found some hidden correlations, which supports the authors opinion that we need to consider time lags to analyze correlations of software metrics.

Comments:

The authors said "we propose an exploratory analysis method ... so that you can study software evolution without an explicit goal" in the introduction. However, what this paper do is to take into account the existence of time lag in correlation analysis. This will improve the traditional correlation analysis, but I couldn't catch why considering time lags contributes to study software evolution *without an explicit goal*. Do you mean that correlation analyses, not limited to the proposed technique, can contribute to such an objective? If so, why such correlation analyses can contribute the objective? Further descriptions about this will be helpful for me.

How should practitioners use the proposed technique? In this study, the authors checked all the possible combinations of variables and parameters. Is it the supposed way in which practitioners use the proposed technique? If so, I'm afraid that the proposed technique takes a long time to complete its task due to the huge amount of possible combinations.

Minor Comments:

p.1 left column; duplicate statement "in step after the delay ..."

p.2 right column; wrong citation "[Isida2008]"

p.2 right column; I'm wondering that "N(N-1)/2 * 12^3" should be "N(N-1)/2 * 13^3" because the range of each kind of parameters is from 0 to 12, in which 13 values exist.

p.3 left column; Why do data points for the explanatory variable range from 0 to 11, not from 0 to 12? Please explain it.

p.3 left column; "put together into a single data point. *A* variable of the explanatory variable ..."

p.3 right column; "Note that regular correlation analysis is *included* in DCA since ..."

p.3 right column; "# of committed changes" is redundant.

p.4 right column and p.5 left column; there exist some citations for table 3, but I guess they should be for table VII.

p.5 left column; "it can be seen that DCA *extracts* the correlation at the correct intervals."

p.5 left column; "In this paper, we implement *a* case study ..."

p.5 right column; "it is revealed unexpected *events*"

p.5 right column; "A proposal method can assist for *researcher* and project manager"

p.5 the section of DISCUSSION; the section seems to lack some articles on nouns or forget to use the plural form.

PAPER: 11 TITLE: An Exploratory Analysis for Studying Software Evolution: Time-Delayed Correlation Analysis AUTHORS: Yosuke Yamatani and Masao Ohira

OVERALL EVALUATION: 0 (borderline paper) REVIEWER'S CONFIDENCE: 4 (high)

I fundamentally disagree with the idea of randomly looking for strong correlations. Software Engineering datasets are much too large to do this, there will be all kinds of meaningless relationships that you’ll find. Understand the process then look at the data. That being said, I like how you get around the stationary property of CCF by using DCA. I don’t understand your results and your discussion is weak.

At first, I didn’t understand how your delayed correlation was different from the standard cross-correlation. As you point out with CCF you need a stationary process and software development is not stationary, for example, release #1 might take two weeks, while release #2 might take 4 weeks. Since the release date is not regular, user bugs will sometimes occur after two weeks and sometimes after four weeks, so the correlation at 2 and 4 weeks will be lower than it would be if we grouped release that took 2 and releases that took 4 weeks independently. The authors DCA technique effectively looks at each correlation independently and only keeps the strong correlations.

I don’t understand the results that you have. How can you tell us anything without first telling us the lag which had the highest correlations? Perhaps you could tell us the variance in correlation at each lag? You can’t just take the top correlation and tell us there’s a strong correlation. When you have such large datasets there will be random strong relationships between almost all variables. The discussion is not a discuss and the results are much to hard to understand.