![[PukiWiki]](image/wadanyan.png "[PukiWiki]")
This paper presented an approach to automatically cluster files with license statements that are unable to be detected by existing rule-based license identification tools. The authors used grep and hierarchical clustering to group these files. The approach was applied on 2838 unknown license files of Debian version 7.8.0.
Points in favour
This paper tackles an interesting problem which is both relevant in research and practice. The research problem is well motivated. The paper is well-structured and the writing is clear.
Points against
In the grep step, to identify the BSD licenses, the authors used the discriminant conditions and the flow conditions shown in Figure 2. There are four clauses in that figure. It is not clear to me what do those clauses mean (e.g., what is "neither the name"? what is "all advertising materials"?). Please briefly describe them in the paper.
In the discussion, the authors explained the rationale behind the not "so high" precision (0.89). But they did not explain the reasons behind the even lower recall (0.64).
The authors calculated the precision/recall/F-value for the grep step. For the clustering step, the authors used another set of metrics (ratio of SLC, ratio of SF, etc.). However, these metrics are not standard performance evaluation metrics. It would be useful to show the actual precision/recall/F-value just as the grep step.
This paper proposes a method to more efficiently check the licenses in software projects by building on a regular-expression-based license identification tool, and cluster together similar unknown licenses for human perusal. Overall, I think that the method seems to be practically useful and would make a good contribution to the workshop.
One major issue with the paper, however, is the clarity of the technical description. Specifically, in Section III.c. the alpha and beta conditions were a bit difficult to understand. For condition alpha it is easy enough to understand by looking at the equation, but what is the particular motivation behind the heuristic threshold in Equation (2). Why does it make sense? The beta condition was quite difficult to understand even after several readings. This seems to be a little bit like the "exchange algorithm" in the following paper, is this correct?
Algorithms for bigram and trigram word clustering Sven Martin, Joerg Liermann, Hermann Ney Speech communication
There were quite a few typos (e.g. "sometimes sometimes", "hight", "pourpose") that could be caught either with a spell checker or a careful read of the paper. I'd suggest using a spell checker and reading through the paper again carefully. It might also be a good idea to use a grammar checker such as the one built in to Microsoft Word. Also, it is not "a data", but "a datum" or "a piece of data". Also, I believe it's usually called F-measure, not F-value?
It would also be interesting to use a more supervised approach to machine learning to identify code with different licenses. For example, every time a user annotates a particular cluster of licenses, this annotated data could be used to train and improve a machine learning model.
The paper describes an approach to automatically infer license rules from license descriptions marked as UNKNOWN by the Ninka tool. These new license rules could then be used to improve the effectiveness of the Ninka tool (that require a predefined set of license rules to be able to identify source code licenses). To date, Ninka can identify 112 licenses. The proposed approach consists in three steps. The Ninka tool is applied on source code files and files with license statements reported as UNKNOWN by Ninka are clustered. Next, each cluster is analysed to extract patterns in the license descriptions. Finally the obtained patterns are used to generate license rules. However, this paper elaborates only on the first step of the proposed approach.
Overall, the paper addresses an interesting problem. The proposed approach could improve the precision/recall of rule based tools like Ninka and FOSSology. However, certain aspects of the work needs to be clarified. For example the authors states that « The GPL/BSD-related licenses could be distinguishable not by using rigid combinations of regular expressions but by finding license names and versions including “or later” options in license statements. » This statement is not entirely true since a licensor could add some conditions and―or exceptions to an existing license. For example, in the case of BSD3, by adding four more sentences to it we get Apache Licences v1.1. How does this affect the accuracy of the proposed approach?
Some Typos:
clusters which are are suitable -> clusters which are suitable a large number of licenses sometimes sometimes must be checked to be reused -> a large number of licenses sometimes must be checked to be reused For the GLP license family -> For the GPL license family Pourpose-> Purpose
