Alhazmi, M. and Altahhan, A. orcid.org/0000-0003-1133-7744 (2023) Best Fit Activation Functions for Attention Mechanism: Comparison and Enhancement. In: Proceedings of 2023 International Joint Conference on Neural Networks (IJCNN). 2023 International Joint Conference on Neural Networks (IJCNN), 18-23 Jun 2023, Gold Coast, Australia. IEEE ISBN 9781665488679
Abstract
Activation functions are one of the critical elements of neural networks that allow them to produce non-linear, fine and complex decision boundaries. Yet, their effects are not very well understood in the context of attention mechanisms. In this paper, we investigate the role of two widely used family of activation functions in conjunction with three attention mechanisms on two widely used image classification models; ResNet50 and MobileNetV2. We modified the structures of these classification models by infusing them with three attention mechanisms, CBAM, BAM, and Triplet Attention. In addition, we equipped them with different activation functions, including ReLU, ELU, and a newly proposed activation function that we call ELU+. The resultant models' performances were examined in the domain of facial expression recognition using three datasets; two lab-controlled, CK+ and JAFFE, and one real-world, FER2013. Compared with the baseline models, our results show a significant increase of up to +30% of models' performance when using the newly proposed AF.
Metadata
Item Type: | Proceedings Paper |
---|---|
Authors/Creators: |
|
Copyright, Publisher and Additional Information: | © 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. |
Dates: |
|
Institution: | The University of Leeds |
Academic Units: | The University of Leeds > Faculty of Engineering & Physical Sciences (Leeds) > School of Computing (Leeds) |
Depositing User: | Symplectic Publications |
Date Deposited: | 11 Oct 2023 15:32 |
Last Modified: | 11 Oct 2023 15:33 |
Published Version: | https://ieeexplore.ieee.org/document/10191885 |
Status: | Published |
Publisher: | IEEE |
Identification Number: | 10.1109/ijcnn54540.2023.10191885 |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:203741 |