Time Series Prediction Using Support Vector Machines, the Orthogonal and the Regularised Orthogonal Least Squares Algorithms

Generalisation properties of support vector machines, orthogonal least squares and other variants of the orthogonal least squares algorithms are studied in this paper. In particular the zero-order regularised orthogonal least squares algorithm that has been proposed in (Chen et al. 1996) and the first order regularised orthogonal least squares algorithm which can be obtained using the cost function support vector machines will be discussed. Simple noisy sine and sinx functions are used to show that overfitting in the orthogonal least squares algorithm can be greatly reduced if the free parameters of the algorithm are selected properly. Results on three chaotic time series show that the orthiogonal least squares algorithm is slightly inferior compared to the other three algorithms. However, the strength of the orthogonal least squares algorithm lies in the ability to obtain a very concise or parsimonious model and the algorithm has the fewest number of free parameters compared to the other algorithms.