Mathematical Programming Potential Functions: New Algorithms for Nonlinear Function Approximation

Linear and quadratic-programming perceptrons for regression are new potential function methods for nonlinear function approximation. Potential function perceptrons, which have been proposed by Aizerman and colleagues in the early 1960's work in the following way: in the first stage patterns from the training set are mapped into a very high dimensional linearisation space by performing a high dimensional non-linear expansion of training vectors into the so-called linearisation space. In this space the perceptron's design function is determined. In the algorithms proposed in this work a non-linear prediction function is constructed using linear-or quadratic-programming routines to optimize the convex cost function. In Linear Programming Machines the L1 loss function is minimised, while Quadratic Programming Machines allow the minimisation of the L2 cost function, or a mixture of both the L1 and L2 noise models. Regularisation is implicitly performed by the expansion into linearisation space by choosing a suitable kernel function), additionally weight decay regularisation is available. First experimental results for one-dimensional curve-fitting using linear programming machines demonstrate the performance of the new method.