A Machine Translation System Evaluation Standard Under The Guidance Of Qiu Maoru’s Translation Equivalence Formula

Since machine translation(MT) was realized on computers, machine translation evaluations(MTE) have been conducted both manually and automatically to assess the properties of machine translation systems(MTS). Some researchers, based on their own experience, have demonstrated the advantages and disadvantages of MTS, which not only provides MTS users with methods as to how to use MTS, but also provides the developers some suggestions as to how to update the system. However, these human evaluations are made with no reference to a certain standard, thus they are rather descriptive and subjective. Other researchers and programmers have invented BLEU, NIST and METEOR and many other metrics to automatically calculate the average quality of the machine translations. Although these automatic evaluations of machine translations have solve the problems such as the high cost and time consuming that manual evaluation dose have, their evaluation methods have just determined the translation quality based on the strings and their degree of similarity; hence their evaluation reliability is still doubtful.To overcome the shortcomings of the previous research of machine translation evaluation and to increase MTS users’operation efficiency, this thesis attempts to establish a MTS evaluation standard through which the abilities and disabilities of a MTS can be objectively and comprehensively disclosed. To guarantee its comprehensiveness and systematicness, the equivalence vs. translation equivalence formula proposed by Prof. Qiu Maoru is employed as the basis of this standard, while adhering to the particularities of MTS.This thesis begins with a general review of the previous MTE studies in the first chapter. The following chapter mainly introduces Qiu’s translation theory. In the third chapter, based on Qiu’s formula and with reference to the particularities of MTS, an objective and comprehensive evaluation standard is established to define the process of evaluation and provide the interpretation of the test results. This evaluation process is, in the fourth chapter, automated into a software-Equivalence Evaluator. The paper concludes that:an evaluation made according to this standard can reveal whether the machine translation system has the ability to achieve the equivalence of all levels just as listed in Qiu’s formular, thereby not only helping the users to find the best system of translation quality, but also to discover which system is more suitable for their own purpose. In addition, the tests show that this software can quantitively evaluate the ability of a MTS at all levels and also successfully compare multiple systems.