The role of the loss function in the probabilistic function approximation

Generalising results on the time series estimation it is natural to consider function approximation with finite data observations in a probabilistic setting. The function is treated as a stochastic process where for each point in the functions domain the function is a random variable. Equivalently the function can be considered as a single random variable whose range is a space of functions. In this paper two results well known within the context of time series estimation and stochastic control are generalised to probabilistic function approximation problems. Under mild conditions on the space of functions it is shown that the optimal function estimate corresponds for all reasonable symmetrical loss functions to the pointwise conditioned expectation given the observed data. Further in the case where the space of functions belongs to the class of Gaussian process the optimal estimate is the conditional expectation even for asymmetric loss functions.