Analysis and implications of a negative parameter in Tikhonov regularisation

The application of Tikhonov regularisation to the least squares (LS) problem arises frequently in machine learning, for example, in regression and the calculation of the excess risk (out-of-sample prediction error) from a given set of noisy observations. It requires the minimisation with respect to x of a function f(x, λ), where λ is the regularisation parameter. If λ ≥ 0, there exists an optimal value λopt of λ such that the vector x(λopt) that minimises f(x, λ) is numerically stable and its error with respect to x(0) is small. It has been claimed that λopt may be negative, and the aim of this paper is the analysis of the consequences of this condition. It is shown theoretically that the condition λ < 0 yields a family of solutions x(λ), each of whose members has a large error and is unstable. Furthermore, the L-curve, which is a method for the calculation of the value of λopt, yields a good result for λ ≥ 0, and it also shows that λ < 0 yields unsatisfactory solutions. The L-curve implies, therefore, that λopt ≥ 0, which is in accord with the theoretical analysis. Examples of LS problems that consider λ < 0 and λ ≥ 0 are shown, and the unsatisfactory results for λ < 0 are evident.