An Exploratory Study Of TEM-4 Reading Task Difficulty

As the only large-scale nationwide standardized test for English majors at the foundation stage in China so far,TEM-4(Test for English majors,Grade 4) is administered by the national foreign language teaching advisory committee under the higher education department of the Ministry of Education.With the Teaching Syllabus for English Majors being the basic guiding principle,and the English sub-committee of the national foreign language teaching advisory committee being in charge of test construction,Tem-4 is a criterion-referenced test aimed at the evaluation of the English teaching quality in China. The purpose of TEM-4 is to promote the implementation of the teaching syllabus,to measure English majors' ability to use English,so as to contribute to improving the English teaching quality in China.One of the primary purposes of a test is to provide necessary information of the test-takers for decision-making.In order to be scientific,unbiased and authoritative,a test is supposed to be able to test what it purports to measure.Thus,one of the challenges facing test designers and test users who are concerned with gauging the influence of task characteristics on candidate performance is how to determine the difficulty of tasks.A greater understanding of the factors affecting task difficulty can assist in the choice of a suitable range of tasks for assessment purposes and also has the potential to influence the way levels of test performance are described.It is now well understood that aspects of test task di(?)ficulty can have an important effect on test performance and it would thus seem imperative to incorporate information about test task difficulty explicitly into the design of language tests and,more importantly,into the interpretation of test scores.But test task difficulty study is a much neglected topic.In this research,identifying the variables which uniquely account for significant variance in the percent correct obtained by examinees for each item in the TEM-4 reading comprehension part is a major focus.This dissertation tries to explore the relationship between reading task difficulty and reading task characteristics,with a view to forming a better understanding of the nature of the TEM-4 reading task.The principal aims of the present study are:1) To identify key task characteristics and task conditions that are most likely to affect the difficulty of TEM-4 reading tasks.2) To investigate the impact on test scores from systematically varying task characteristics and task conditions and,in cases where clear effects are noted,to explore possible reasons for differences in task difficulty;to figure out the factor structure or potential relationship among these characteristics,and to identify the acco untability of item difficulty in the TEM-4 reading section by the set of factors specified in this research.The research materials are the eight reading passages and the corresponding 40 items on the 2005 and 2006 TEM-4 papers.More than 300,000 test-takers' test scores are processed for the item difficulty index.A rating instrument with 143 test task characteristic variables is constructed,including propositional content variables(subject matter,genre,vocabulary and other aspects),organizational characteristics variables(rhetorical features of patterns, grammar,rhetorical organization,coherence,cohesion),pragmatic characteristics variables (speech acts and language functions,register),length and other language-related variables (number of words,different types of sentences,negations and frontings).These are performed on three levels:passage,key sentence,and item.What's more,these variables are of three different types:counted,mathematically calculated and rated.The first two types are conducted by the researcher herself,and the rated variables are collected from 10 experts using a specifically designed rating sheet.Multiple techniques are exploited to analyze the raw data.Descriptive statistical analysis, reliability of ratings,standardization of variables,pair-wise correlation,exploratory factor analysis,confirmatory factor analysis and multiple regressions are all employed.The results indicate that 22 reading task characteristics variables are found responsible for the TEM-4 reading task difficulty,among which passage variables take up a far more important share than key sentence variables and item variables.Three salient constructs are discovered,namely,passage complexity factor,passage length factor and key sentence salience factor,and they account for 31.2%of the variation in the reading task difficulty.The conclusion part summarizes the answers tO the research questions and points out the theoretical,methodological implications and implications for language testing practices. The limitations and future research suggestions are mentioned in the end.