Radial Basis Function Network Configuration Using Mutual Information and the Orthogonal Least Squares Algorithm

Input nodes of neural networks are usually predetermined by using a priori knowledge or selected by trial and error. For example, in pattern recognition applications the input nodes are usually the given pattern features and in system identification applications the past input and output data are often used as inputs to the network. Some of the input variables may be irrelevant to the task in hand and therefore, may cause a deterioration in the network and consume expensive computation time. In the present study, the mutual information between the input variables and the output of the network is used to select a suboptimal set of input variables for the network. The variables are selected according to the information content relevant to the output. Variables which have a higher mutual information with the output and lower dependence on other selected variables are used as network inputs. The algorithms are derived based on heuristics and performance is assessed by using radial basis function (RBF) networks trained with the orthogonal least squares algorithm (OLS), which selects the hidden layer nodes of the network according to the error reduction ratios on the network output. Both real and simulated data sets are used to demonstrate the effectiveness of the new algorithms.