| In the field of IR(Information Retrieval),the evaluation of IR systems plays an important role. The evaluation of IR systems has significant impact and effect on researchs, development and applications within the field of IR. Test Collection become the main form of the evaluation of IRsystems, dating roughly from the 1960s.the quality of the test collection is considered to be a key factor of the correctness of the results of such evaluation.Test Collection is still in wide use now.The language form which is the part of Test Colletcion decide the area of evaluation.most of the Tect Collection which have been installed abroad use the english language.the english Tect Collection can not take effect bath on the Chinese information retrieval system.the research of chinese Test Colletion is in its beginning stage. a lack of small chinese Test Colletion restrict the development of IR greatly. If a chinese Test Collection can be set up in laboratory environment, which will undoubtedly be able to greatly facilitate the use of researchers, thereby speeding up the research work.According to the framework of existing foreign test collections,this paper put forward a proposal for building the evaluation of IR systems. The following contributions are achieved in this paper:(1) Construct a Small Chinese test collection includes three parts, that is, document set, queries and relevance judgment set. The document set collects and arranges 1648 papers which mainly about Chinese content; according to the main content of the document set primarily construct 15 query topics; integrate the relevant judge results from two pairs of judge members and construct the relevant judgment set in order to evaluate retrieval system Performance.(2)Adopt Kappa coefficient of concordance and Kendall coefficient of concordance two different kinds of statistics to analyze the consistency by the two pairs of relevant judgment results, the final judge is that the two results has highly consistency, which indicates the highly reliability of the test set. |
PDF Full Text Request