Churcher, G, Atwell, ES and Souter, DC (1997) A generic template to evaluate integrated components in spoken dialogue systems. In: Hirschberg, J, Kamm, C and Walker, M, (eds.) Proceedings of ACL/EACL'97 Workshop on Interactive Spoken Dialog Systems: bringing speech and NLP together in real applications. ACL/EACL'97 Workshop on Interactive Spoken Dialog Systems: bringing speech and NLP together in real applications, 11-12 Jul 1997, The National Distance Education University (UNED), Madrid, Spain. Association for Computational Linguistics , 9 - 16.
Abstract
We present a generic template for spoken dialogue systems integrating speech recognition and synthesis with 'higher-level' natural language dialogue modelling components. The generic model is abstracted from a number of real application systems targetted at very different domains. Our research aim in developing this generic template is to investigate a new approach to the evaluation of Dialogue Management Systems. Rather than attempting to measure accuracy/speed of output, we propose principles for the evaluation of the underlying theoretical linguistic model of Dialogue Management in a given system, in terms of how well it fits our generic template for Dialogue Management Systems. This is a measure of 'genericness' or 'application-independence' of a given system, which can be used to moderate accuracy/speed scores in comparisons of very unlike DMSs serving different domains. This relates to (but is orthogonal to) Dialogue Management Systems evaluation in terms of naturalness and like measurable metrics (eg Dybkjaer et al 1995, Vilnat 1996, EAGLES 1994, Fraser 1995); it follows more closely emerging qualitative evaluation techniques for NL grammatical parsing schemes (Leech et al 1996, Atwell 1996).
Metadata
Item Type: | Proceedings Paper |
---|---|
Authors/Creators: |
|
Editors: |
|
Copyright, Publisher and Additional Information: | (c) ACL, 1997. Reproduced with permission from the publisher. |
Keywords: | Evaluation; comparisons; generic model; standards |
Dates: |
|
Institution: | The University of Leeds |
Academic Units: | The University of Leeds > Faculty of Engineering & Physical Sciences (Leeds) > School of Computing (Leeds) > Artificial Intelligence & Biological Systems (Leeds) |
Depositing User: | Symplectic Publications |
Date Deposited: | 13 Jan 2015 11:25 |
Last Modified: | 21 Feb 2024 13:36 |
Published Version: | http://www.aclweb.org/anthology/W97-0602 |
Status: | Published |
Publisher: | Association for Computational Linguistics |
Identification Number: | 10.3115/1641462.1641464 |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:82277 |