Winkler, J. orcid.org/0000-0002-4629-8928, Mitrouli, M. and Koukouvinos, C. (2022) The application of regularisation to variable selection in statistical modelling. Journal of Computational and Applied Mathematics, 404. 113884. ISSN 0377-0427
Abstract
The aim of variable selection is the identification of the most important predictors that define the response of a linear system. Many techniques for variable selection use a constrained least squares (LS) formulation in which the constraint is imposed in the 1-norm (the lasso), or the 2-norm (Tikhonov regularisation), or a linear combination of these norms (the elastic net). It is always assumed that a constraint must necessarily be imposed, but the consequences of its imposition have not been addressed. This assumption is considered in this paper and it is shown that the correct application of Tikhonov regularisation to the overdetermined LS problem min kAx − bk2 requires that A and b satisfy a condition C. If this condition is satisfied, then the solution of the LS problem with this constraint is numerically stable and the regularisation error e between the solution of this problem and the solution of the LS problem is small. If, however, the condition C is not satisfied, then the error e is large. The condition C is derived from a refined normwise condition number of the solution of the LS problem. The paper includes examples of regularisation and variable selection with correlated variables that illustrate the theory in the paper.
Metadata
Item Type: | Article |
---|---|
Authors/Creators: |
|
Copyright, Publisher and Additional Information: | © 2021 Elsevier B.V. This is an author produced version of a paper subsequently published in Journal of Computational and Applied Mathematics. Uploaded in accordance with the publisher's self-archiving policy. Article available under the terms of the CC-BY-NC-ND licence (https://creativecommons.org/licenses/by-nc-nd/4.0/). |
Keywords: | Ridge regression (Tikhonov regularisation); condition number; discrete Picard condition; regularisation error; lasso |
Dates: |
|
Institution: | The University of Sheffield |
Academic Units: | The University of Sheffield > Faculty of Engineering (Sheffield) > Department of Computer Science (Sheffield) |
Depositing User: | Symplectic Sheffield |
Date Deposited: | 21 Oct 2021 11:43 |
Last Modified: | 26 Oct 2022 00:13 |
Status: | Published |
Publisher: | Elsevier |
Refereed: | Yes |
Identification Number: | 10.1016/j.cam.2021.113884 |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:179180 |