Milner, R. and Hain, T. orcid.org/0000-0003-0939-3464 (2016) Segment-oriented evaluation of speaker diarisation performance. In: 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Proceedings of the 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 20-25 Mar 2016, Shanghai, China. IEEE ISBN 978-1-4799-9988-0
Abstract
High performance diarisation is a necessity for a variety of applications, and the task has been studied extensively in the context of broadcast news and meeting processing. Upon introduction of the task in NIST led evaluations, diarisation error rate (DER) was introduced as the standard metric for evaluation, and it has been consistently used to compare systems ever since. DER is a frame based metric that does not penalise for producing many short segments. However, practical systems that require diarisation input are typically not able to cope well with such artefacts. In this paper we illustrate the need for an alternative metric focussing on segments, instead of duration or boundaries only. We propose a segment based F-measure, which specifically addresses issues such as reference errors, matching start and end boundaries, and speaker pairing. The performance of the metric is analysed in the context of state-of-the-art systems and compared with other existing metrics. It is shown to give a deeper insight into the segmentation quality over the standard metrics, and thus better value for to understand impact on follow on tasks such as ASR.
Metadata
Item Type: | Proceedings Paper |
---|---|
Authors/Creators: |
|
Copyright, Publisher and Additional Information: | © 2016 IEEE |
Dates: |
|
Institution: | The University of Sheffield |
Academic Units: | The University of Sheffield > Faculty of Engineering (Sheffield) > Department of Computer Science (Sheffield) |
Depositing User: | Symplectic Sheffield |
Date Deposited: | 21 Jul 2017 13:37 |
Last Modified: | 21 Jul 2017 13:37 |
Published Version: | https://doi.org/10.1109/ICASSP.2016.7472721 |
Status: | Published |
Publisher: | IEEE |
Refereed: | Yes |
Identification Number: | 10.1109/ICASSP.2016.7472721 |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:107042 |