Gardner, P., Lord, C. orcid.org/0000-0002-2470-098X and Barthorpe, R. orcid.org/0000-0002-6645-8482 (2019) A unifying framework for probabilistic validation metrics. Journal of Verification, Validation and Uncertainty Quantification, 4 (3). 031005. ISSN 2377-2158
Abstract
Probabilistic modeling methods are increasingly being employed in engineering applications. These approaches make inferences about the distribution for output quantities of interest. A challenge in applying probabilistic computer models (simulators) is validating output distributions against samples from observational data. An ideal validation metric is one that intuitively provides information on key differences between the simulator output and observational distributions, such as statistical distances/divergences. Within the literature, only a small set of statistical distances/divergences have been utilized for this task; often selected based on user experience and without reference to the wider variety available. As a result, this paper offers a unifying framework of statistical distances/divergences, categorizing those implemented within the literature, providing a greater understanding of their benefits, and offering new potential measures as validation metrics. In this paper, two families of measures for quantifying differences between distributions, that encompass the existing statistical distances/divergences within the literature, are analyzed: f-divergence and integral probability metrics (IPMs). Specific measures from these families are highlighted, providing an assessment of current and new validation metrics, with a discussion of their merits in determining simulator adequacy, offering validation metrics with greater sensitivity in quantifying differences across the range of probability mass.
Metadata
Item Type: | Article |
---|---|
Authors/Creators: |
|
Copyright, Publisher and Additional Information: | © 2019 ASME. This is an Open Access article distributed under the terms of the Creative Commons Attribution Licence (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. |
Dates: |
|
Institution: | The University of Sheffield |
Academic Units: | The University of Sheffield > Faculty of Engineering (Sheffield) > Department of Mechanical Engineering (Sheffield) |
Funding Information: | Funder Grant number ENGINEERING AND PHYSICAL SCIENCE RESEARCH COUNCIL (EPSRC) EP/N010884/1 EUROPEAN REGIONAL DEVELOPMENT FUND UNSPECIFIED ENGINEERING AND PHYSICAL SCIENCE RESEARCH COUNCIL (EPSRC) EP/J013714/1 ENGINEERING AND PHYSICAL SCIENCE RESEARCH COUNCIL (EPSRC) EP/R006768/1 |
Depositing User: | Symplectic Sheffield |
Date Deposited: | 28 Nov 2019 15:51 |
Last Modified: | 25 Nov 2020 01:40 |
Status: | Published |
Publisher: | American Society of Mechanical Engineers |
Refereed: | Yes |
Identification Number: | 10.1115/1.4045296 |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:147356 |