We propose a principled framework for learning with infinitely many features, situations that are usually induced by continuously parametrized feature extraction methods. Such cases occur for instance when considering Gabor-based features in computer vision problems or when dealing with Fourier features for kernel approximations. We cast the problem as the one of finding a finite subset of features that minimizes a regularized empirical risk.
After having analyzed the optimality conditions of such a problem, we propose a simple algorithm which has the flavour of a column-generation technique.
We also show that using Fourier-based features, it is possible to perform approximate infinite kernel learning. Our experimental results on several datasets show the benefits of the proposed approach in several situations including texture classification and large-scale kernelized problems involving about thousand examples.
While several recent works address the problem of automatic feature generation Bengio , most of the time these features are still manually crafted based on specific domain knowledge. For instance, in several machine vision applications, features are extracted by appropriate preprocessing of images. One widely used feature extraction method is the Gabor filter Serre et al.
Similarly, wavelet decompositions or time-frequency representations are frequent feature extraction methods for signal classification problems Lee et al. One major drawback of these feature generation methods is that they come along with several continuous parameters and thus they can potentially produce infinitely many features.
