NCLT Seminar Series 2006/2007The NCLT seminar series takes place on Wednesdays from 4-5 pm in Room L2.21 (School of Computing).
The schedule of presenters for the 2006/2007 series (Semester 2) is as follows:
In this talk I will present an evaluation of four approaches to automatic grammaticality judgements. Such judgements can be used to automatically grade essays or to trigger a computationally expensive error analysis. The first approach follows the traditional view that the grammar determines grammaticality. The test corpus is parsed with the XLE parser and "starred" sentences are classified as ungrammatical. The second approach is similar. Here we prune a PTB-trained PCFG so that it rejects ungrammatical input. Thirdly, n-gram methods are considered. If a sentence contains an n-gram below a certain frequency threshold, it is rejected. Finally, my own approach (developed in collaboration with Jennifer Foster) is included in the evaluation. The approach compares the actual probability output of a statistical parser with a probability estimated from a reference corpus of grammatical sentences in order to judge the grammaticality.
A long-standing and unresolved issue in the parsing literature is whether parsing less-configurational languages is harder than e.g. parsing English. German is a case in point. Results from Dubey and Keller (2003) suggest that state-of-the-art parsing scores for German are generally lower than those obtained for English, while recent results from Kuebler et al. (2006) raise the possibility that this might be an artifact of encoding schemes and data structures of treebanks, which serve as training resources for probability parsers. In the talk I present new experiments to test this claim. We use the PARSEVAL metric, the Leaf-Ancestor metric as well as a dependency-based evaluation, and present complimentary approaches measuring the effect of controlled error insertion on treebank trees and parser output. We also provide extensive cross-treebank conversion. The results of the experiments show that, contrary to Kuebler et al. (2006), the question whether or not German is harder to parse than English remains undecided.
For reasons that I can't quite recall, I've been the track coordinator for EACL-06 and ACL-07. Given that, I have some interesting (I think!) statistics on the trends from one conference to the other regarding topics of papers, and country of origin of those same papers. I'll then finish with some predictions arising from those figures.
I present the results of several experiments using multi-word units (MWUs) as a means to impose constraints on both probabilistic parsing and surface generation with automatically-acquired (treebank-based) grammars. In the case of surface realisation from LFG f-structures with automatically-acquired treebank-based LFG approximations modest but significant gains in accuracy can be made. Experiments integrating the same MWUs in treebank-based probabilistic parsing yielded smaller, but still statistically significant gains. I analyse the results and offer a number of explanations as to why the gains achieved are smaller than might be naively expected.
I talk about zero pronoun identification in Japanese corpus. Since zero pronouns appear quite often in Japanese texts, identifying them is one of the important issues in Japanese NLP, and is also required in long distance dependency (LDD) resolution at the level of f-structure representation, and in automatic case-frame extraction from a large corpus. I introduce a simple method of zero pronoun identification which uses verbal morphological features which signify transitivity of a verb, along with the probability of the cooccurrence of a verb and nouns which are attached with certain case-marking particles. I will show and analyze the result of applying the method to 500 sentences randomly chosen from Kyoto Text Corpus, and the parsing output of the same sentences by the Japanese dependency parser which does not take zero pronouns into account, and try to explain the advantage and drawback of the method, and possible ways to improve its performance.
Studies on Controlled Language (CL) suggest that by removing certain linguistic features that are known to be problematic for Machine Translation (MT) from a source text, the MT output can be improved. A further assumption is that an improvement in MT output will result in lower post-editing effort. With the ever-increasing emphasis in the translation industry on higher volumes and faster throughput, it is not surprising that this assumption is of interest to those who manage multi-lingual high-volume translation projects. Increasingly, translation service providers are asked to provide post-editing services in addition to their traditional translation/localisation services. The expectation is that post-editing will be faster than human translation and that, therefore, post-editing should not cost as much as translation. However, the assumption that CL reduces post-editing effort has not been tested empirically. It is worthy of closer inspection, not least because CLs can cover a broad range of linguistic features (OBrien 2003). This paper presents results from a study designed to test the assumed link between CL and post-editing effort by measuring the technical, temporal and cognitive post-editing effort (Krings 2001) for English sentences in a user manual that have been translated into German using an MT system and that have been subsequently post-edited by nine professional translators. In this study, the linguistic features known to be problematic for MT are called negative translatability indicators or NTIs for short. The post-editing effort for sentences containing NTIs is compared with the post-editing effort for sentences where all known NTIs have been removed. In addition, relative post-editing effort (Krings 2001) a comparison of post-editing effort and translation effort - is measured. A comparison will be made between NTIs that generate a high-level of post-editing effort and those that generate a lower level of post-editing effort. The methodologies employed include the use of the keyboard monitoring tool, Translog (Jakobsen 1999, Hansen 2002), and Choice Network Analysis (Campbell 1999).
Campbell, Stuart (1999), A Cognitive Approach to Source Text Difficulty in Translation, in Target, 11:1, pp. 33-63.
Hansen, Gyde (ed) (2002), Empirical Translation Studies Process and Product, Copenhagen Studies in Language 27, Copenhagen: Samfundslitteratur.
Jakobsen, Arnt Lykke (1999), Logging Target Text Production with Translog, in Hansen, Gyde (ed), Probing the Process in Translation: Methods and Results, Copenhagen Studies in Language 24, Copenhagen: Samfundslitteratur, pp. 9-20.
Krings, Hans P. (2001), Repairing Texts: Empirical Investigations of Machine Translation Post-Editing Processes, Kent/Ohio: Kent State University Press, edited/translated by G. S. Koby et al.
OBrien, Sharon (2003), Controlling Controlled English An Analysis of Several Controlled Language Rule Sets, in Proceedings of EAMT/CLAW 2003, Dublin: Dublin City University, pp. 105-114.
Data-driven approaches to machine translation (MT) achieve state-of-the-art results. Many syntax-aware approaches, such as Example-Based Machine Translation and Data-Oriented Translation, make use of tree pairs aligned at sub-sentential level. Obtaining sub-sentential alignments manually is time-consuming and error-prone, and requires expert knowledge of both source and target languages. We propose a novel, language pair-independent algorithm which automatically induces alignments between phrase-structure trees. We evaluate the alignments themselves against a manually aligned gold standard, and perform an extrinsic evaluation by using the aligned data to train and test a DOT system. Our results show that translation accuracy is comparable to that of the same translation system trained on manually aligned data, and coverage improves.
Automatic Speech Recognition is based on several components: signal
processor, acoustic model, language model, and search. In this talk, we explore the use
of Random Forests (RFs) in language modeling, the problem of predicting the
next word based on words already seen. The goal is to develop a new language
model smoothing technique based on randomly grown Decision Trees (DTs). This new
technique is complementary to many of the existing techniques dealing with
We present a transparent model for ranking sentences that incorporates topic relevance as well as an aboutness and importance feature. We describe and compare five methods for estimating the importance feature. The two key features that we use are graph-based ranking and ranking based on reference corpora of sentences known to be important. Independently those features do not improve over the baseline, but combined they do. While our experimental evaluation focuses on informational queries about people, our importance estimation methods are completely general and can be applied to any topic.top
Previous studies suggest that the application of Controlled Language (CL) rules can significantly improve the readability, consistency, and machine-translatability of technical documentation. One of the justifications for the application of CL rules is that they can reduce the post-editing effort required to bring Machine Translation (MT) output to acceptable quality. In certain situations, however, post-editing services may not always be a viable solution. Web-based information is often expected to be made available in real-time to ensure that its access is not restricted to certain users based on their locale. Uncertainties remain with regard to the actual usefulness of MT output for such users, as no empirical study has examined the impact of CL rules on the usefulness and comprehensibility of MT technical documents from a Web user's perspective. This presentation focuses on the results of an online experiment conducted at Symantec, a leader in Internet security technology. Using a customer satisfaction questionnaire, a set of machine-translated technical support documents was published and randomly evaluated by genuine French and German users. The findings indicate that the introduction of CL rules can have a significant impact on the comprehensibility of German MT documents.top
C-Structures and F-Structures for the British National Corpus Joachim Wagner, Djamé Seddah, Jennifer Foster and Josef van Genabith
A Comparative Evaluation of Deep and Shallow Approaches to the Automatic Detection of Common Grammatical Errors Joachim Wagner, Jennifer Foster and Josef van Genabith
The objective of this talk is to explain the prohibition on two determiners in genitive noun phrases in Irish using the frameworks of the Minimalist Program and Distributed Morphology. I will first give a brief overview of Generative Syntax, the Minimalist Program and Distributed Morphology. This will be followed with a recap of previous work on Irish noun phrases involving the DP Hypothesis. I will then introduce the notion of feature valuation in Distributed Morphology which includes a particular view of nominalisation. These concepts provide the framework for an elegant explanation of Determiner-Noun agreement, Genitive case assignment and Definiteness agreement. The prohibition on two determiners in genitive noun phrases in Irish follows naturally from this explanation.top
|Last update: 1st October 2010|