The Generalized Bayesian Committee Machine

In this paper we introduce the Generalized Bayesian Committee Machine (GBCM) for applications with large data sets. In particular, the GBCM can be used in the context of kernel based systems such as smoothing splines, kriging, regularization networks and Gaussian process regression which —for computational reasons— are otherwise limited to rather small data sets. The GBCM provides a novel and principled way of combining estimators trained for regression, classification, the prediction of counts, the prediction of lifetimes and other applications which can be derived from the exponential family of distributions. We describe an online version of the GBCM which only requires one pass through the data set and only requires the storage of a matrix of the dimension of the number of query or test points. After training, the prediction at additional test points only requires resources dependent on the number of query points but is independent of the number of training data. We confirm the good scaling behavior using real and experimental data sets.