![[PukiWiki]](image/wadanyan.png "[PukiWiki]")
#author("2019-03-04T11:27:36+09:00","","") #author("2019-03-04T11:52:22+09:00","","") [[福井/査読結果]] *査読結果 [#t9942130] **REVIEW 1 [#pb040f03] Summary:~ Expert finding for question answering is an important and challenging problem in community-based question answering (CQA) systems. This paper presents an improved tag-based approach based on PMF (Probabilistic Matrix Factorization) and term expansion using word embedding. The novelty is the ability to address similar tags (e.g., JAVA and JAVA-9). The paper is well written. However, one main problem is that this paper cannot convince readers that the performance improvement is substantial. My main comments are as follows. Is the tag-based method a good choice? ~ The main expert finding approaches can be categorized into two groups: the authority-oriented approach, and the topic-oriented approaches. The authority-oriented expert finding methods are based on link analysis for the ask-answer relation between users in the rating matrix (e.g., [1][2]). The topic-oriented expert finding methods are based on latent topic modeling techniques for the content of the questions (e.g., [3][4]). The paper proposed a tag-based method that belongs to the authority-oriented method. The proposed method regards the user’s expertise by tags. The expertise of the users was translated to (user, tag) scores. However, the method relies heavily on the quality of the tags. If a new question is tagged with an inappropriate tag, the method will fail. Lack of discussion or comparison on state-of-the-art studies. ~ This paper only considers one baseline, which is published in 2014 on ASONAM. However, in terms of this problem, many following studies proposed more advanced approaches. This paper did not discuss them, such as [4][5][6][7]. The result is not substantial.~ The performance in terms of nDCG is 0.833 (baseline is 0.824). The improvement looks too minor to prove the effectiveness.~ [1] H. Zhu, H. Cao, H. Xiong, E. Chen, and J. Tian, “Towards expert finding by leveraging relevant categories in authority ranking,” in Proc. 20th ACM Int. Conf. Inform. Knowl. Manage., 2011, pp. 2221– 2224.~ [2] H. Zhu, E. Chen, H. Xiong, H. Cao, and J. Tian, “Ranking user authority with relevant knowledge categories for expert finding,” World Wide Web, vol. 17, pp. 1–27, 2013.~ [3] Tian Y, Kochhar P S, Lim E P, et al. Predicting best answerers for new questions: An approach leveraging topic modeling and collaborative voting[C]//International Conference on Social Informatics. Springer, Berlin, Heidelberg, 2013: 55-68.~ [4] Wang J, Sun J, Lin H, et al. Convolutional neural networks for expert recommendation in community question answering[J]. Science China Information Sciences, 2017, 60(11): 110102.~ [5] Zhao Z, Zhang L, He X, et al. Expert finding for question answering via graph regularized matrix completion[J]. IEEE Transactions on Knowledge and Data Engineering, 2015, 27(4): 993-1004.~ [6] Wang X, Huang C, Yao L, et al. A survey on expert recommendation in community question answering[J]. Journal of Computer Science and Technology, 2018, 33(4): 625-653.~ [7] Tondulkar R, Dubey M, Desarkar M S. Get me the best: predicting best answerers in community question answering sites[C]//Proceedings of the 12th ACM Conference on Recommender Systems. ACM, 2018: 251-259.~ **REVIEW2 [#c693713c] This paper describes an improved approach for expert recommendation based on PMF (Probabilistic Matrix Factorization) and term expansion for Q&A services such as Stack Overflow. The paper later compares an existing approach with the proposed approach through a preliminary experiment to show that the proposed approach provides better recommendation accuracy. -Strengths of the paper: --The authors provide a description of existing approaches and how they add to them in this study. -Major issues: --As far as I can tell, the paper is not really about bots. In fact, bots don't get mentioned almost at all in the paper. I do realize that the authors plan to implement the recommendation system as a bot, however, even this is not discussed in the paper (beyond being mentioned once). At the very least, I'd expect the authors to provide a discussion on why implementing it as a bot would be beneficial, and what aspects of bots this method would leverage. Perhaps this paper would better fit in something like Mining Software Repositories (MSR) venue. --To emphasize the previous point, even the experiment in section 4, clearly states the core aspect of the paper "the existing approach (PMF) and our approach (PMF + term expansion) are compared using the recommendation accuracy." --I honestly don't think this paper would this workshop (BOTSE 2019), perhaps the next paper that includes a bot component to it would fit better. This does not to mean that the proposed approach for improving recommendation accuracy is invalid. -Minor issues: --There are phrasing issues in the paper. Some weird sentences ("an important issue has been arising"), some missing words ("In Stack Overflow, users can vote answers"). Typically this is a minor issue, but it was hindering my ability to understand the study described in the paper. -- I think in a short paper like this, there is no need for the last paragraph of the introduction (starting with "In what follows..."). I would use this space in a better way. --"In the study, PMF..." - in which study? --References should not be used as a structural part of a sentence (e.g., ". In [12]..."). --In the results, the authors show that based on nDGC scores their approach outperforms the existing method. Is it marginally better or is this difference significant? There is no discussion of this. --There could be additional related work discussed (on the challenges of ranking SO questions and the issue of unanswered questions. On expert search, etc...) **REVIEW3 [#n70cc469] This paper presents a method for expert finding in a community based question-answering scenario (e.g., Stack Overflow). This method could be used by a bot that would help focus the attention of expert users on questions they have expertise on.~ The paper itself is interesting and easy to follow - although the authors should probably do a good bit of proofreading.~ The main concerns i have about this paper are:~ -first of all, it does not present a bot as such - i.e., the elements of design and implementation of a bot based on the expertise finding mechanism are not presented. A bot is (or should be) more than just a search algorithm and making a bot would be more than just taking your algorithm and putting it in a bot platform. To some extent, the fact that this paper does not present a real bot is not real an issue - but the paper should at least describe how the authors would create a bot based on their algorithm - and that is not an easy task. -expert finding is also a task that seems to me more than just finding who has expertise in a particular domain. The availability of the experts, their interest in a topic etc. are probably important elements to consider as well. Otherwise the "experts" won't really have a good experience in the system. Checking the literature, I could find multiple papers on the topic addressing participants' interest, using for instance variations of economic models. This is something the authors should consider to make a paper on expertise finding for a community QA system viable.
