| Lexical diversity encompasses a wide variety of measures, all of which seek to quantify the range of vocabulary deployed in a given text. Researchers use these measures of lexical diversity in many fields, including language acquisition, stylistics, neuropathology, and linguistic forensics. Unfortunately, text length confounds all the measures, leading to questions of the conclusions of some studies. Many alternative measures have been introduced but all have failed to overcome the problem of text length correlation. We introduce and test a new measure of lexical diversity: the measure of textual, lexical diversity (MTLD). We test MTLD and 13 of the best known traditional measures of lexical diversity against the largest corpus yet established for such a test: 23 genres of spoken and written texts, comprising 414,000 words. The results of these tests supply evidence that none of the traditional measures avoid correlation with text length. MTLD, however, does not correlate with text length over the ranges tested suggesting that MTLD is the first reliable measure of lexical diversity. The significance of such a measure is that researchers and educators will be able to assess the lexical diversity of both spoken and written texts without concern for the differing text lengths. We also test all the traditional measures against a further corpus of NS and NNS. In these tests, both MTLD and some of the traditional measures predicted differences in the results. We conclude that MTLD is the only sophisticated measure that avoids correlation with text length but that using other sophisticate measures, in conjunction with MTLD, may be the best approach to analysing texts. |
