Haq, F.U., Shin, D. orcid.org/0000-0002-0840-6449 and Briand, L. (2022) Efficient online testing for DNN-enabled systems using surrogate-assisted and many-objective optimization. In: Proceedings of the 44th International Conference on Software Engineering. 44th International Conference on Software Engineering (ICSE '22), 21-29 May 2022, Pittsburgh, Pennsylvania, USA. Association for Computing Machinery , pp. 811-822. ISBN 9781450392211
Abstract
With the recent advances of Deep Neural Networks (DNNs) in real-world applications, such as Automated Driving Systems (ADS) for self-driving cars, ensuring the reliability and safety of such DNN-enabled Systems emerges as a fundamental topic in software testing. One of the essential testing phases of such DNN-enabled systems is online testing, where the system under test is embedded into a specific and often simulated application environment (e.g., a driving environment) and tested in a closed-loop mode in interaction with the environment. However, despite the importance of online testing for detecting safety violations, automatically generating new and diverse test data that lead to safety violations presents the following challenges: (1) there can be many safety requirements to be considered at the same time, (2) running a high-fidelity simulator is often very computationally-intensive, and (3) the space of all possible test data that may trigger safety violations is too large to be exhaustively explored. In this paper, we address the challenges by proposing a novel approach, called SAMOTA (Surrogate-Assisted Many-Objective Testing Approach), extending existing many-objective search algorithms for test suite generation to efficiently utilize surrogate models that mimic the simulator, but are much less expensive to run. Empirical evaluation results on Pylot, an advanced ADS composed of multiple DNNs, using CARLA, a high-fidelity driving simulator, show that SAMOTA is significantly more effective and efficient at detecting unknown safety requirement violations than state-of-the-art many-objective test suite generation algorithms and random search. In other words, SAMOTA appears to be a key enabler technology for online testing in practice.
Metadata
Item Type: | Proceedings Paper |
---|---|
Authors/Creators: |
|
Copyright, Publisher and Additional Information: | © 2022 The Author(s). |
Keywords: | Bioengineering; Clinical Research |
Dates: |
|
Institution: | The University of Sheffield |
Academic Units: | The University of Sheffield > Faculty of Engineering (Sheffield) > Department of Computer Science (Sheffield) |
Depositing User: | Symplectic Sheffield |
Date Deposited: | 13 Jan 2023 10:39 |
Last Modified: | 13 Jan 2023 10:39 |
Published Version: | http://dx.doi.org/10.1145/3510003.3510188 |
Status: | Published |
Publisher: | Association for Computing Machinery |
Refereed: | Yes |
Identification Number: | 10.1145/3510003.3510188 |
Related URLs: | |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:194819 |