| There are no interval marks between words in Chinese. As the first job in any Chi-nese natural language processing (NLP), Chinese word segmentation maps the sequence of characters into a sequence of words. Traditional algorithms, such as forth maximum matching, deal with this problem depend on the lexicon, and must maintain it manually. In this paper, we present a new algorithm to deal with this issue and try to get an acceptable precision by using the minimum hand-crafted resource.In this paper, we introduce a new algorithm in the third chapter. Our algorithm im-proves traditional statistical algorithm in two aspects:first, we introduce the log-likelihood ratio (LLR) algorithm to quantity the relations between single Chinese characters, and com-bine this algorithm with pointwise mutual information (PMI) and information entropy (IE) to obtain an unified algorithm to quantity this relations; second, we extend the be-gram language model to tri-gram, and optimization this language model to avoid exponential increase of complexity of the model.We present an application-specific word segmentation framework in the fourth chap-ter, using the statistical algorithm we present before to detect unknown words and adapt the out of base segmentor to different application-specific standards. |
