Orthogonal random features approximate a Bessel kernel, offering sharper bounds than random Fourier features.
problem Approximating Gaussian kernel efficiently for large datasets.
method Use of Haar orthogonal matrices to construct orthogonal random features and analyze their bias and variance.
result Orthogonal random features approximate a Bessel kernel, not the Gaussian kernel, with sharper bounds.
We present Random Partition Kernels, a new class of kernels derived by demonstrating a natural connection between random partitions of objects and kernels between those objects. We show how the construction can be used to create kernels from methods that would not normally be viewed as random partitions, such as Random…
New random feature maps for Laplacian and related kernels.
problem Challenges in approximating the Laplacian kernel and its generalizations.
method Developed random feature maps for Laplacian and related kernels, providing efficient sampling schemes.
result Demonstrated the efficacy of these random feature maps on real datasets.
Improved kernel ridge regression for large datasets using weighted random binning.
problem Efficiently approximating kernel matrices for large-scale datasets.
method Introduced weighted random binning features for locality sensitive hashing.
result Weighted random binning features generate Gaussian processes of any desired smoothness.
A faster graph kernel using optical random features.
problem High computation cost of graphlet kernel due to isomorphism test.
method Kernel random features, optical random features, mean kernel metric.
result The proposed method is orders of magnitude faster with similar or better accuracy.
Unified view on random walk and Weisfeiler-Leman kernels, improving accuracy.
problem Improving graph kernel methods for better classification accuracy.
method Define and analyze walk-based node refinement methods, relate to Weisfeiler-Leman test, and introduce new walk-based kernels.
result Walk-based kernels are as expressive as Weisfeiler-Leman subtree kernel but support non-strict neighborhood comparison.
Positive-definite kernel functions are fundamental elements of kernel methods and Gaussian processes. A well-known construction of such functions comes from Bochner's characterization, which connects a positive-definite function with a probability distribution. Another construction, which appears to have attracted less…
Kernel ridgeless regression with random features shows good generalization without explicit regularization.
problem Generalization of kernel ridgeless regression without explicit regularization.
method Investigation of ridgeless regression with random features and stochastic gradient descent, exploring the effect of random features error and spectral density optimization.
result Random features error exhibits the double-descent curve, leading to improved generalization.
Improves efficiency of random feature approximations for dot product kernels.
problem Efficiency of random feature approximations for dot product kernels.
method Generalization of existing random feature approximations using complex-valued random features, theoretical analysis of variances, data-driven optimization approach.
result Complex-valued random features can significantly reduce the variances of approximations.
New method reduces variance and bias in approximating indefinite kernels.
problem Approximating non-stationary indefinite kernels with low variance and bias.
method Generalized orthogonal random features (GORF)
result GORF achieves lower variance and approximation error compared to existing methods.
A new method for approximating softmax and Gaussian kernels with reduced error.
problem Approximating softmax and Gaussian kernels with low error.
method Simplex Random Features (SimRFs) and SimRFs+.
result SimRFs provide the smallest MSE among weight-independent geometrically-coupled PRF mechanisms.
HRFs adaptively linearize kernels for accurate approximations.
problem Linearizing softmax and Gaussian kernels for machine learning applications.
method Generalizes Bochner's Theorem for kernels, uses random features for compositional kernels.
result Strong theoretical guarantees and unbiased approximation with smaller relative errors.
New graph kernel scales well with graph size and number, achieving state-of-the-art performance.
problem Graph kernels lose structure information when representing graphs.
method Proposes a positive-definite global alignment graph kernel using random features and random graph embeddings.
result Achieves quasi-linear scalability with respect to graph size and number.
New spectral mixture representation for isotropic kernels simplifies random Fourier features.
problem Applying Random Fourier Features to complex kernels.
method Decompose isotropic kernels into scale mixtures of α-stable random vectors.
result Constructive spectral sampling formula for various kernels.
Kernel methods are powerful and flexible approach to solve many problems in machine learning. Due to the pairwise evaluations in kernel methods, the complexity of kernel computation grows as the data size increases; thus the applicability of kernel methods is limited for large scale datasets. Random Fourier Features (R…
We introduce the Mondrian kernel, a fast random feature approximation to the Laplace kernel. It is suitable for both batch and online learning, and admits a fast kernel-width-selection procedure as the random features can be re-used efficiently for all kernel widths. The features are constructed by sampling trees via a…
A novel algorithm for unbiased graph kernel estimation with subquadratic time complexity.
problem Efficient estimation of graph kernels for large networks.
method Random walk-based algorithm with modulation function parameterized by neural network.
result Higher-quality kernel estimates and efficient scalable learning on larger networks.
Devoted to multi-task learning and structured output learning, operator-valued kernels provide a flexible tool to build vector-valued functions in the context of Reproducing Kernel Hilbert Spaces. To scale up these methods, we extend the celebrated Random Fourier Feature methodology to get an approximation of operator-…
New method improves Gaussian kernel approximations for high-frequency data.
problem Limited scalability of kernel-based models to large data sets.
method Local random feature approximations using Maclaurin expansions and polynomial sketches.
result Significant improvement in kernel approximations and downstream performance for high-frequency data.
Random features improve neural operators' generalization properties.
problem Improving generalization of neural operators.
method Unified framework for spectral regularization techniques and operator-valued kernels.
result Established optimal learning rates and required number of neurons.
No-trick kernel adaptive filtering uses deterministic features for scalability and robustness.
problem Scalability issues in kernel methods for large datasets.
method Deterministic feature-map construction using polynomial-exact solutions.
result Deterministic features outperform random Fourier features in performance and scalability.
Criterion extends identifiability for continuous mixtures of kernels.
problem Identify continuous mixtures of kernels.
method Generating-function accessibility criterion based on moment-generating functions or Laplace transforms.
result Criterion applies to mixtures of discrete and continuous variables.
New findings on kernel regression in the quadratic regime, improving understanding of machine learning models.
problem Understanding kernel ridge regression in the quadratic asymptotic regime.
method Extended study of kernel regression to the quadratic regime, establishing approximation bounds and spectral distributions.
result Broad class of inner-product kernels exhibit behavior similar to a quadratic kernel, with precise asymptotic training and test errors characterized.
Random Fourier features is one of the most popular techniques for scaling up kernel methods, such as kernel ridge regression. However, despite impressive empirical results, the statistical properties of random Fourier features are still not well understood. In this paper we take steps toward filling this gap. Specifica…
SRF improves kernel approximation and GP regression performance.
problem Efficient kernel approximation and Bayesian kernel learning in large-scale regression problems.
method Stein variational gradient descent to generate high-quality random features and approximate spectral measure posteriors.
result SRF outperforms traditional approaches in kernel approximation and GP regression.
The study examines Fisher information matrices and neural tangent kernels for simple ReLU networks with random weights.
problem Understanding the relationship between Fisher information matrices and neural tangent kernels for 2-layer ReLU networks.
method Analyzes Fisher information matrices and neural tangent kernels for 2-layer ReLU networks with random hidden weights, focusing on spectral decomposition and eigenfunctions.
result Obtained an approximation formula for functions represented by 2-layer neural networks.
Lecture notes on kernel functions and Random Fourier Features.
problem Understanding and approximating kernel functions in machine learning.
method Mathematical background and proofs of concentration results.
result Estimation of error in Random Fourier Features approximation.
Novel Newton method for large-scale kernel methods using random features.
problem Efficiently solving large-scale finite-sum minimization problems in RKHS.
method Randomized feature-based Newton method for empirical risk minimization.
result Local superlinear and global linear convergence of the method.
Optimizes differentially private kernel learning with random projection.
problem Privacy-preserving learning algorithms with optimal performance.
method Differentially private kernel ERM algorithm based on random projection in reproducing kernel Hilbert space.
result Achieves minimax-optimal excess risk rates for various loss functions.
In this paper, we compare 5 different nonlinear kernels: min-max, RBF, fRBF (folded RBF), acos, and acos-χ2, on a wide range of publicly available datasets. The proposed fRBF kernel performs very similarly to the RBF kernel. Both RBF and fRBF kernels require an important tuning parameter (γ). Interestingly, for a …
A new method slices and sums radial kernels faster.
problem Fast computation of large kernel sums in kernel methods.
method Random projections to 1D subspaces and QMC for selecting projections.
result QMC-slicing outperforms existing methods on test datasets.
The approximation of nonlinear kernels via linear feature maps has recently gained interest due to their applications in reducing the training and testing time of kernel-based learning algorithms. Current random projection methods avoid the curse of dimensionality by embedding the nonlinear feature space into a low dim…
Development of metrics for structural data-generating mechanisms is fundamental in machine learning and the related fields. In this paper, we give a general framework to construct metrics on random nonlinear dynamical systems, defined with the Perron-Frobenius operators in vector-valued reproducing kernel Hilbert space…
New methods improve Reservoir Computing for chaotic time series prediction.
problem Chaotic time series prediction in Reservoir Computing.
method Established Recurrent Kernel limit, introduced Structured Reservoir Computing.
result Structured Reservoir Computing is faster and more memory-efficient.
Random Machines improves SVM performance with free kernel choice.
problem Efficiency and accuracy in solving classification and regression problems.
method Bagged-weighted support vector model with free kernel choice.
result Improved accuracy and reduced computational time.
Kernel methods are an extremely popular set of techniques used for many important machine learning and data analysis applications. In addition to having good practical performances, these methods are supported by a well-developed theory. Kernel methods use an implicit mapping of the input data into a high dimensional f…
Develops RKHS framework for analyzing tree ensembles.
problem Analyzing the theoretical properties of tree ensembles.
method Reproducing Kernel Hilbert Spaces (RKHS) for tree ensembles.
result Characterizes Random Forest predictor as minimizer of a penalized empirical risk functional in RKHS.
This paper improves kernel quantile regression with random features for handling heavy-tailed noises.
problem Handling heavy-tailed noises in kernel quantile regression.
method Introduces a refined error decomposition and establishes a novel connection between KQR-RF and KRR-RF.
result Establishes capacity-dependent learning rates for KQR-RF under mild conditions on the number of random features, which are minimax optimal up to some logarithmic factors.
End-to-end kernel learning using generative RFFs for improved performance.
problem Improving kernel learning performance and generalization.
method Develops a generative network via RFFs to implicitly learn the kernel, followed by a linear classifier, jointly trained by ERM.
result Shows superior generalization performance over classical methods in real-world tasks.
Paper extends RPD for better handling multiple modalities and non-convexity.
problem Handling multiple modalities and non-convexity in data clouds.
method Computes RPD in a reproducing kernel Hilbert space using kernel principal component analysis.
result The method outperforms RPD and is comparable to other models on benchmark datasets.
Random feature approximation speeds up spectral methods and improves learning rates.
problem Improving the efficiency and generalization of spectral methods in large-scale algorithms.
method Combining random feature approximation with spectral regularization methods.
result Optimal learning rates for estimators over various regularity classes, including those not in the RKHS.
New method approximates complex kernel norms with random features, making learning tractable.
problem Complexity of learning with kernel methods in high dimensions.
method Random features approximations to Fp norms, focusing on p>1. result For p>1, the number of random features required is polynomial in the sample size, making learning tractable. RFFNet scales kernel methods to large datasets by learning kernel relevance.
problem Scaling kernel methods to large datasets while maintaining interpretability.
method Designs random Fourier features for ARD kernels and uses first-order stochastic optimization for learning kernel relevances.
result RFFNet achieves low prediction error and identifies relevant features, leading to more interpretable solutions.
We consider the problem of improving kernel approximation via randomized feature maps. These maps arise as Monte Carlo approximation to integral representations of kernel functions and scale up kernel methods for larger datasets. Based on an efficient numerical integration technique, we propose a unifying approach that…
Random Forest kernels improve performance in various regression and survival tasks.
problem Improving performance of Random Forest in high-dimensional data with noisy features.
method Developed and evaluated data-driven RF kernels for regression, classification, and survival tasks.
result RF kernels are competitive or superior to RF in most scenarios, especially for survival tasks.
New RFs reduce kernel approximation variance and improve Transformer performance.
problem Efficient approximation of Gaussian and softmax kernels for kernel methods and Transformers.
method Parameterized, positive, non-trigonometric RFs optimized for variance reduction.
result Significant variance reduction in practice, outperforming previous methods.
Study reveals an equivalence principle for the spectrum of random inner-product kernel matrices in polynomial scaling.
problem Understanding the spectrum of random kernel matrices in polynomial scaling regimes.
method Investigates random matrices with nonlinear kernel functions applied to inner products of uniformly distributed vectors.
result The spectrum of the random kernel matrix is asymptotically equivalent to a simpler matrix model through free additive convolution.
Deep kernel learning improves performance on complex tasks.
problem Limited flexibility of classical kernel methods for complicated tasks.
method Random Fourier Features (RFF) integrated into deep architecture.
result Significantly boosts flexibility and richness of kernel machines.