Drummond, R. orcid.org/0000-0002-2586-1718, Turner, M.C. orcid.org/0000-0003-2161-7635 and Duncan, S.R. orcid.org/0000-0002-9525-7305 (2024) Reduced-order neural network synthesis with robustness guarantees. IEEE Transactions on Neural Networks and Learning Systems, 35 (1). pp. 1182-1191. ISSN 2162-237X
Abstract
In the wake of the explosive growth in smartphones and cyber-physical systems, there has been an accelerating shift in how data are generated away from centralized data toward on-device-generated data. In response, machine learning algorithms are being adapted to run locally on board, potentially hardware-limited, devices to improve user privacy, reduce latency, and be more energy efficient. However, our understanding of how these device-orientated algorithms behave and should be trained is still fairly limited. To address this issue, a method to automatically synthesize reduced-order neural networks (having fewer neurons) approximating the input-output mapping of a larger one is introduced. The reduced-order neural network's weights and biases are generated from a convex semidefinite program that minimizes the worst case approximation error with respect to the larger network. Worst case bounds for this approximation error are obtained and the approach can be applied to a wide variety of neural networks architectures. What differentiates the proposed approach to existing methods for generating small neural networks, e.g., pruning, is the inclusion of the worst case approximation error directly within the training cost function, which should add robustness to out-of-sample data points. Numerical examples highlight the potential of the proposed approach. The overriding goal of this article is to generalize recent results in the robustness analysis of neural networks to a robust synthesis problem for their weights and biases.
Metadata
Item Type: | Article |
---|---|
Authors/Creators: |
|
Copyright, Publisher and Additional Information: | © 2022 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other users, including reprinting/ republishing this material for advertising or promotional purposes, creating new collective works for resale or redistribution to servers or lists, or reuse of any copyrighted components of this work in other works. Reproduced in accordance with the publisher's self-archiving policy. |
Keywords: | Neural network compression; reduced order systems; robustness |
Dates: |
|
Institution: | The University of Sheffield |
Academic Units: | The University of Sheffield > Faculty of Engineering (Sheffield) > Department of Automatic Control and Systems Engineering (Sheffield) |
Depositing User: | Symplectic Sheffield |
Date Deposited: | 12 Feb 2024 16:10 |
Last Modified: | 12 Feb 2024 20:18 |
Status: | Published |
Publisher: | Institute of Electrical and Electronics Engineers (IEEE) |
Refereed: | Yes |
Identification Number: | 10.1109/tnnls.2022.3182893 |
Related URLs: | |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:209077 |