Statistical modelling of MT output corpora for information extraction

The output of state-of-the-art machine translation (MT) systems could be useful for certain NLP tasks, such as Information Extraction (IE). However, some unresolved problems in MT technology could seriously limit the usability of such systems. For example robust and accurate word sense disambiguation, which is essential for the performance of IE systems, is not yet achieved by commercial MT applications. In this paper we try to develop an evaluation measure for MT systems that could predict their possible usability for some IE tasks, such as scenario template filling, or automatic acquisition of templates from texts. We focus on statistically significant words for a text in a corpus, which are used now for some IE tasks such as automatic template creation (Collier, 1998). Their general importance for IE was also substantiated by our material, where they often include name entities and other important candidates for filling IE templates. We suggest MT evaluation metrics which are based on comparing the distribution of statistically significant words in corpora of MT output and in human reference translation corpora. We show that there are substantial differences in such distributions between human translations and MT output, which could seriously distort IE performance. We compare different MT systems with respect to the proposed evaluation measures and look into their relation to other MT evaluation metrics. We also show that the statistical model suggested could highlight specific problems in MT output that are related to conveying factual information. Dealing with such problems systematically could considerably improve the performance of MT systems and their usability for IE tasks.