| Two transformations of the text stream are introduced, the bigram proximity matrix (BPM) and the trigram proximity matrix (TPM). In this dissertation, the question of semantic preservation via these structures is answered through a series of supervised learning, unsupervised learning and hypothesis testing experiments.; The TDT Pilot Corpus was used as the source of text units (newscasts). After transforming each newscast to a BPM and TPM, semantic similarity was assessed using thirteen measures. Supervised learning experiments were conducted on the full dimensionality of the proximity matrices as determined by the lexicon, and further experiments were performed on reduced text by means of denoised and stemmed versions. Correct classification ratios ranged from 0.85 to 0.99 using kNN classification.; Another test of the capacity of the BPM and TPM to capture and preserve meaning was performed in a series of hypothesis tests. A graph-theoretic approach was used for the tests, where the test statistic was the number of edges crossing topic boundaries. Overwhelmingly, the null hypotheses were rejected, indicating clear differentiation of topics.; Unsupervised learning experiments were conducted after dimensionality reduction using Isometric Feature Mapping. Model-based clustering (MBC) was then used to cluster documents. A visualization aid, ReClus, was created to examine the results from MBC. Results indicated the presence of latent topics. Other multidimensional visualization aids, like parallel coordinates were used to corroborate these results.; Further supervised learning experiments were conducted using Bayesian classifiers. Results of these reduced dimensionality experiments compared favorably to those performed on the full-dimensionality space.; A further test of the BPM and TPM was devised in the form of topic change detection. Results of experiments with 50 and 416 documents provided convincing evidence of the capacity of the BPM and TPM to capture and preserve semantics.; In summary, these novel transformations of the text stream, the BPM and TPM, capture and preserve meaning. The resultant structures are amenable to efficient computational methods for classification, clustering, and topic change determination. |
