Research On International Large Scale Mathematics Evaluation

In the late twentieth century, with the awareness of human capital in the socio-economic development, countries or regions increasingly concerned about the human capital cultivation nationally or locally, and paid attention to the educational process and the quality and efficiency of the educational system. International large-scale assessments were formed and developed. At the same time, mathematics as a fundamental and necessary tool to discipline for modern scientific and technological development, has become one of the basic qualities of modern talent of economic development. In consideration of the great similarity in mathematics course from various countries, the International Large-Scale Mathematical Assessments(ILSMA)are increasingly got attention by different countries, and present unique style compared with the other international large-scale assessment.TIMSS and PISA, which are developed by International Education Association(IEA) and the Organizational for Economic Cooperation and Development(OECD) respectively, have become an international platform, giving feedback and comparing the quality and efficiency of mathematical education among different counties and regions, and helping the policy making and policy adjustment. Therefore, it aroused our curiosity of researching on the large-scale international assessment of mathematics.Mathematics is the main domain of PISA 2012. In the same year, the outcomes of Shanghai students ranked the first once again after 2009. To our surprise, all of the 1953 students that attended the PISA 2012 as a sample take the Unified Mathematics Academic Assessment for Junior High School(UMAAJHS). The correlation coefficient of scores of two assessments was 0.71. The difference between them and influencing factors are drawing our great attention.Based on the statement above, this paper analyzes the internal system construction of international large-scale mathematical assessment, and from this perspective, enquires the reasons that make a big difference between ILSMA and UMAAJHS, and the necessity of the construction of the evaluation system of UMAAJHS, which is also our research objective.Firstly, this paper takes research on the system construction and the characteristics of ILSMA. In order to facilitate this study, this paper gives an understanding of evidence-based inference based on Mislevy, and combines evidence-centered assessment system design with the assessment triangle9 makes cognition! observation! and interpretation! proposed by National Research Institute(NRC) as the three dimensions of the large-scale comparative evaluation of mathematical analysis. Then this paper takes TIMSS mathematics assessment and PISA mathematics assessment as examples, using historical research, analyzes the longitudinal development of the two assessment systems. The similarities and differences in system construction of the two different assessments are compared, highlighting the internal structural features of the ILSMA.Then we expanded our data analysis with quantitative methods. First, the study tries to explore the effect of the internal system differences between ILSMA and UMAAJHS on the scores of the students who took the PISA 2012 and UMAAJHS, including the math scores and related non-cognitive results. With the analyzes model construction work in analytic model for effect of learning outcomes, this paper makes use of data of influencing factor in the mathematical assessment combined with PISA 2012, including data on mathematics learning opportunities---contents, teaching practice, the quality of teaching, learning mathematics motivation and belief. By using of hierarchical linear model(HLM), influencing factors in UMAAJHS and ILSMA scores differences at the individual student and school level respectively are analysized. Quantitative results show that different large-scale mathematics assessments learning have different impact on construction of analytic model for effect of learning outcome. This means we need to establish our own analytic model for effect of learning outcome or quality evaluation and analysis system.The main results and conclusion are presented as below.(1) the development of different ILSMA share similarity, which show similarities not only in the research and policy-driven fusion, but also in target analysis framework, the impact on the establishment of framework of analytic model for effect of learning outcome and methods of techniques analysis, as well as in the development stage---from the objective analysis framework, the establishment of content dimension ruler, hierarchical description ruler, to establishment of cognitive process ruler, the consistency of the whole process.(2) the basis of construction of large-scale mathematical assessments is the internal systematical consistency. The three elements of evaluation systems: cognition!--- objective analytical framework of assessment, observation!--- a blueprint for assessment, using of the item types, and coding system design, interpretation!--- the formation of the ruler, using of statistical models, the division of benchmark or proficiency level. They play different roles in assessment systems, and they are interdependence and mutual restraint, even correcting each other, forming the internal consistency of the entire large-scale mathematical assessments, ensuring the reliability and validity of the survey results. Based on internal consistency, the ILSMA also reflects the comprehensive evaluation system---the integrated use of technology and methods, and the coherence---the comparability of the survey results in different years is established.(3) Significant effect of the assessment framework on the evaluation purposes. For different evaluation purposes, it should reflect what has been known and what has been done(TIMSS mathematics assessment, for example), or should reflect what should be known and how to do(PISA mathematics assessment, for example). TIMSS shows the learning outcomes and performance levels in form of pure knowledge or clues, while PISA emphases on key competence, and show the learning outcomes by problem-solving. Different ILSMA carried out according to different mathematical assessment target framework, and even got the different results in the observation---distribution of the question type and quantities, as well as interpretation---the differences on the construction of the ruler.(4) The variety in different dimensions within the assessment framework of ILSMA will affect the evaluation results. According to the design of content and cognitive processes in the ILSMA, and PISA mathematics assessment as the example, UMAAJHS was compared with PISA mathematics assessment on the content and process dimensions. And the interpretation rate of contents difference between PISA mathematics assessment and UMAAJHS scores is 36.7%, while the interpretation ratio of process dimension difference between them is 37.3%. The later which promotes a unit has a greater impact on UMAAJHS than the former. Cognitive! component embodied in assessment framework. But in view of its unifying nature on the other elements, the example above shows the inherent differences in ILSMA system construction, and even the difference in items distribution at the same dimension will affect the results of large-scale mathematics assessment.(5) The influencing factors of different large-scale mathematics assessment achievement are different. For the same group, when they are studied by different assessment, the influencing factors share similarities and have differences as well, and even have appreciable effect. For different mathematical assessment results, there exists an intermediary between the effects of different factors. Especially for mathematics self-efficacy and math work ethic, the former have a significant impact on PISA mathematics assessment outcomes and test scores in UMAAJHS, while the latter only have a significant impact on UMAAJHS. Meanwhile, the latter showed a significant effect for mediation in UMAAJHS, which shows the unique factors of learning outcomes in UMAAJHS. All of these indicate that we should establish our own learning results analysis model and quality evaluation analysis system.In view of the above findings, from comparing the existing performance in UMAAJHS, we find that there are shortcomings in the inherent consistency, and interpretation of results is almost missing. This will greatly affect the test validity and reliability of UMAAJHS results, and influence the definition of junior high school students$ academic level and the feedback on the teaching. Finally, this paper analyze the system of UMAAJHS, mainly on the establishment of the system outlook and target framework, and in infrastructure of UMAAJHS, such as establishment of key competence standards and mathematical problem solving evaluation frameworks, the selection of mathematics target frame content and process-related aspects, in order to help enhance the construction of internal consistency of UMAAJHS, and interpretation of the results, even provide a reference for the reform of UMAAJHS.