arXiv research

Orthogonal random features approximate a Bessel kernel, offering sharper bounds than random Fourier features.

Random feature models approximate functions in Banach spaces efficiently.

Improves efficiency of random feature approximations for dot product kernels.

We study the approximation properties of random ReLU features through their reproducing kernel Hilbert space (RKHS). We first prove a universality theorem for the RKHS induced by random features whose feature maps are of the form of nodes in neural networks. The universality result implies that the random ReLU features…

We propose semi-random features for nonlinear function approximation. The flexibility of semi-random feature lies between the fully adjustable units in deep learning and the random features used in kernel methods. For one hidden layer models with semi-random features, we prove with no unrealistic assumptions that the m…

HARFE approximates sparse additive functions using random features and ridge regression.

A new method for approximating softmax and Gaussian kernels with reduced error.

Random feature approximation speeds up spectral methods and improves learning rates.

Improved kernel ridge regression for large datasets using weighted random binning.

A novel model uses ODE-based random features to model nonlinear dynamical systems.

HRFs adaptively linearize kernels for accurate approximations.

Random Fourier features is one of the most popular techniques for scaling up kernel methods, such as kernel ridge regression. However, despite impressive empirical results, the statistical properties of random Fourier features are still not well understood. In this paper we take steps toward filling this gap. Specifica…

New method improves Gaussian kernel approximations for high-frequency data.

Survey on random features for kernel approximation, focusing on algorithms, theory, and practical applications.

Two ANOVA-based algorithms boost random Fourier feature models for function approximation.

Kernel approximation using randomized feature maps has recently gained a lot of interest. In this work, we identify that previous approaches for polynomial kernel approximation create maps that are rank deficient, and therefore do not utilize the capacity of the projected feature space effectively. To address this chal…

New method reduces variance and bias in approximating indefinite kernels.

New RFs reduce kernel approximation variance and improve Transformer performance.

RQMC improves kernel-based learning by reducing deterministic error and offering computational advantages.

We present an intriguing discovery related to Random Fourier Features: in Gaussian kernel approximation, replacing the random Gaussian matrix by a properly scaled random orthogonal matrix significantly decreases kernel approximation error. We call this technique Orthogonal Random Features (ORF), and provide theoretical…

Random feature method approximates operators with theoretical guarantees and reduced computation.

We consider the problem of improving kernel approximation via randomized feature maps. These maps arise as Monte Carlo approximation to integral representations of kernel functions and scale up kernel methods for larger datasets. Based on an efficient numerical integration technique, we propose a unifying approach that…

Kernel methods are powerful and flexible approach to solve many problems in machine learning. Due to the pairwise evaluations in kernel methods, the complexity of kernel computation grows as the data size increases; thus the applicability of kernel methods is limited for large scale datasets. Random Fourier Features (R…

Random feature model approximates PDE solutions efficiently.

Devoted to multi-task learning and structured output learning, operator-valued kernels provide a flexible tool to build vector-valued functions in the context of Reproducing Kernel Hilbert Spaces. To scale up these methods, we extend the celebrated Random Fourier Feature methodology to get an approximation of operator-…

Kernel methods are powerful learning methodologies that allow to perform non-linear data analysis. Despite their popularity, they suffer from poor scalability in big data scenarios. Various approximation methods, including random feature approximation, have been proposed to alleviate the problem. However, the statistic…

SRF improves kernel approximation and GP regression performance.

Kernel method has been developed as one of the standard approaches for nonlinear learning, which however, does not scale to large data set due to its quadratic complexity in the number of samples. A number of kernel approximation methods have thus been proposed in the recent years, among which the random features metho…

Poor approximators found in neural networks and random feature models.

New random feature maps for Laplacian and related kernels.

Extends random feature analysis to spectral methods and improves learning rates.

Hermite polynomials improve private data generation by reducing feature count.

Random features and KRR generalize similarly when N is large enough.

This paper surveys various methods for dimensionality reduction and nearest neighbor search.

Random features approach has been widely used for kernel approximation in large-scale machine learning. A number of recent studies have explored data-dependent sampling of features, modifying the stochastic oracle from which random features are sampled. While proposed techniques in this realm improve the approximation,…

New quantization methods improve accuracy of Random Fourier Features.

Large-scale kernel approximation is an important problem in machine learning research. Approaches using random Fourier features have become increasingly popular [Rahimi and Recht, 2007], where kernel approximation is treated as empirical mean estimation via Monte Carlo (MC) or Quasi-Monte Carlo (QMC) integration [Yang …

Kernel methods offer the flexibility to learn complex relationships in modern, large data sets while enjoying strong theoretical guarantees on quality. Unfortunately, these methods typically require cubic running time in the data set size, a prohibitive cost in the large-data setting. Random feature maps (RFMs) and the…

Although kernel methods are widely used in many learning problems, they have poor scalability to large datasets. To address this problem, sketching and stochastic gradient methods are the most commonly used techniques to derive efficient large-scale learning algorithms. In this study, we consider solving a binary class…

New method approximates complex kernel norms with random features, making learning tractable.

Random features are improved by variance-reducing couplings, enhancing machine learning models.

Random features provide a practical framework for large-scale kernel approximation and supervised learning. It has been shown that data-dependent sampling of random features using leverage scores can significantly reduce the number of features required to achieve optimal learning bounds. Leverage scores introduce an op…

We find a deterministic equivalent for random feature regression's test error, independent of feature map dimension.

SRMD uses random features for efficient time-frequency analysis.

RFAD uses random features to speed up dataset distillation.

Graph neural networks (GNNs) are powerful machine learning models for various graph learning tasks. Recently, the limitations of the expressive power of various GNN models have been revealed. For example, GNNs cannot distinguish some non-isomorphic graphs and they cannot learn efficient graph algorithms. In this paper,…

Ensembles dynamic models using random feature approximations.

We prove that, under low noise assumptions, the support vector machine with $N\ll m$ random features (RFSVM) can achieve the learning rate faster than $O(1/\sqrt{m})$ on a training set with $m$ samples when an optimized feature map is used. Our work extends the previous fast rate analysis of random features method from…