![[PukiWiki]](image/wadanyan.png "[PukiWiki]")
[[ヤマタニ]] *IWESEP2014＠中之島 [#oe44080c] &attachref(); &attachref(IWESEP2014 参加報告.pptx); **title [#n1f1fc94] An Exploratory Analysis for Studying Software Evolution: Time-Delayed Correlation Analysis **abstruct [#he514af2] We propose an exploratory analysis method for software repositories so that you can study software evolution without an explicit goal. The method extends the traditional correlation analysis to consider a time lag between any pair of metrics. Using data collected from three kinds of repositories, the method automatically calculates optimal parameters to deal with a time lag between metrics and returns the highest correlation coefficient for each pair of metrics. Our case study using one hundred metrics data from three OSS projects shows that the method allows us to find unexpected relationships in some pairs of metrics if we only use the traditional correlation analysis and that using such relationships can be used to build an analysis goal to understand software evolution. **review [#z4f9a824] ----------------------- REVIEW 1 --------------------- PAPER: 11 TITLE: An Exploratory Analysis for Studying Software Evolution: Time-Delayed Correlation Analysis AUTHORS: Yosuke Yamatani and Masao Ohira OVERALL EVALUATION: 1 (weak accept) REVIEWER'S CONFIDENCE: 2 (low) ----------- REVIEW ----------- This paper proposes a method for an exploratory analysis of software evolution, called delayed correlation analysis (DCA). DCA extends the correlation analysis to consider a time gap between any pair of metrics, and returns the highest correlation coefficient for each pair of metrics. With various metrics, the authors conducted a case study and reported some detailed results. Pros: - The problem of the correlation analysis of two time series with gaps should be interesting. I found some Q&A forms and papers. It should be more beneficial if the paper clarifies the differences with such previous approaches. http://stats.stackexchange.com/questions/7727/how-to-correlate-two-time-series-with-gaps-and-different-time-bases http://stats.stackexchange.com/questions/23993/how-to-correlate-two-time-series-with-possible-time-differences http://www.eecs.qmul.ac.uk/~ccloy/project_time_delayed/index.html http://maxi.riken.jp/FirstYear/proceedings/pdf/Scargle_Jeffrey_D.-bin-20110413063416.pdf http://maxi.riken.jp/FirstYear/ppt/O26SCARGLE.pdf http://www.itc.nl/~rossiter/teach/R/R_ts.pdf - The paper studied various metrics from three OSS projects with long histories. Cons: - The result section is not easy to understand. For example, the statements (Section V-A, 2nd paragraph, 4th line and 3rd paragraph, 1st line) are not clear. In addition, I do not understand why the metric pair of # of bugs that severity are blocker and # of commentators is not selected in top correlations. - The proposed method tends to show illusory correlation more compared to the traditional correlation analysis. For an exploratory analysis, this is fine if the users of this method analyze the results carefully. Discussing this issue should be required. Minor comments: - Section I, 3rd paragraph: in step after the delay from the programing development process. in step after the delay from the programing development process. -> in step after the delay from the programing development process. - Section III-B, 3rd paragraph: into a single data point. a value of -> into a single data point. A value of - Section IV-A, last line: grayed part in Figure 2 <- there is no grayed part, C^DCA -> CA^DCA ----------------------- REVIEW 2 --------------------- PAPER: 11 TITLE: An Exploratory Analysis for Studying Software Evolution: Time-Delayed Correlation Analysis AUTHORS: Yosuke Yamatani and Masao Ohira OVERALL EVALUATION: 1 (weak accept) REVIEWER'S CONFIDENCE: 3 (medium) ----------- REVIEW ----------- Summary: This paper proposes a technique to take into account the existence of time lags in correlation analysis of software metrics. The authors conducted an empirical study with three OSS projects and 100 metrics. The authors checked all the possible combinations of the metrics with various setting values, and found some hidden correlations that cannot be found with traditional approaches. Pros: + The considered problem is important. + The approach proposed in this paper makes sense to cope with the considered problem. + The empirical study found some hidden correlations, which supports the authors opinion that we need to consider time lags to analyze correlations of software metrics. Comments: - The authors said "we propose an exploratory analysis method ... so that you can study software evolution without an explicit goal" in the introduction. However, what this paper do is to take into account the existence of time lag in correlation analysis. This will improve the traditional correlation analysis, but I couldn't catch why considering time lags contributes to study software evolution *without an explicit goal*. Do you mean that correlation analyses, not limited to the proposed technique, can contribute to such an objective? If so, why such correlation analyses can contribute the objective? Further descriptions about this will be helpful for me. - How should practitioners use the proposed technique? In this study, the authors checked all the possible combinations of variables and parameters. Is it the supposed way in which practitioners use the proposed technique? If so, I'm afraid that the proposed technique takes a long time to complete its task due to the huge amount of possible combinations. Minor Comments: - p.1 left column; duplicate statement "in step after the delay ..." - p.2 right column; wrong citation "[Isida2008]" - p.2 right column; I'm wondering that "N(N-1)/2 * 12^3" should be "N(N-1)/2 * 13^3" because the range of each kind of parameters is from 0 to 12, in which 13 values exist. - p.3 left column; Why do data points for the explanatory variable range from 0 to 11, not from 0 to 12? Please explain it. - p.3 left column; "put together into a single data point. *A* variable of the explanatory variable ..." - p.3 right column; "Note that regular correlation analysis is *included* in DCA since ..." - p.3 right column; "# of committed changes" is redundant. - p.4 right column and p.5 left column; there exist some citations for table 3, but I guess they should be for table VII. - p.5 left column; "it can be seen that DCA *extracts* the correlation at the correct intervals." - p.5 left column; "In this paper, we implement *a* case study ..." - p.5 right column; "it is revealed unexpected *events*" - p.5 right column; "A proposal method can assist for *researcher* and project manager" - p.5 the section of DISCUSSION; the section seems to lack some articles on nouns or forget to use the plural form. ----------------------- REVIEW 3 --------------------- PAPER: 11 TITLE: An Exploratory Analysis for Studying Software Evolution: Time-Delayed Correlation Analysis AUTHORS: Yosuke Yamatani and Masao Ohira OVERALL EVALUATION: 0 (borderline paper) REVIEWER'S CONFIDENCE: 4 (high) ----------- REVIEW ----------- I fundamentally disagree with the idea of randomly looking for strong correlations. Software Engineering datasets are much too large to do this, there will be all kinds of meaningless relationships that you’ll find. Understand the process then look at the data. That being said, I like how you get around the stationary property of CCF by using DCA. I don’t understand your results and your discussion is weak. At first, I didn’t understand how your delayed correlation was different from the standard cross-correlation. As you point out with CCF you need a stationary process and software development is not stationary, for example, release #1 might take two weeks, while release #2 might take 4 weeks. Since the release date is not regular, user bugs will sometimes occur after two weeks and sometimes after four weeks, so the correlation at 2 and 4 weeks will be lower than it would be if we grouped release that took 2 and releases that took 4 weeks independently. The authors DCA technique effectively looks at each correlation independently and only keeps the strong correlations. I don’t understand the results that you have. How can you tell us anything without first telling us the lag which had the highest correlations? Perhaps you could tell us the variance in correlation at each lag? You can’t just take the top correlation and tell us there’s a strong correlation. When you have such large datasets there will be random strong relationships between almost all variables. The discussion is not a discuss and the results are much to hard to understand.
