This work improves knowledge distillation by transferring full kernel matrices efficiently.
problem Efficiently transferring full pairwise similarity matrices for model compression in deep learning.
method The authors propose a method to transfer the full similarity matrix effectively using the Nyström method, decomposing it into partial matrices.
result The difference between the full kernel matrices of teacher and student can be well bounded by partial matrices, improving optimization efficiency.
Efficient and accurate low-rank approximations of multiple data sources are essential in the era of big data. The scaling of kernel-based learning algorithms to large datasets is limited by the O(n^2) computation and storage complexity of the full kernel matrix, which is required by most of the recent kernel learning a…
New framework estimates eigenvalues of kernel matrices without full matrix construction.
problem Estimating eigenvalues of large kernel matrices efficiently.
method Eigenvalue quantile estimation framework for kernel matrices with quick decay.
result Validates framework with empirical evidence and proves interlacing theorem.
A determinantal point process (DPP) is a probabilistic model of set diversity compactly parameterized by a positive semi-definite kernel matrix. To fit a DPP to a given task, we would like to learn the entries of its kernel matrix by maximizing the log-likelihood of the available data. However, log-likelihood is non-co…
This work improves multi-task regression performance using approximations of full-conformal prediction.
problem Jointly solve multiple regression tasks with better performance than individual tasks.
method Design an approximating prediction region in RKHS for full-conformal multi-task regression.
result The approximating prediction region improves upon split-conformal prediction on synthetic data.
We present memory-efficient and scalable algorithms for kernel methods used in machine learning. Using hierarchical matrix approximations for the kernel matrix the memory requirements, the number of floating point operations, and the execution time are drastically reduced compared to standard dense linear algebra routi…
One approach to improving the running time of kernel-based machine learning methods is to build a small sketch of the input and use it in lieu of the full kernel matrix in the machine learning task of interest. Here, we describe a version of this approach that comes with running time guarantees as well as improved guar…
Kernel principal component analysis (KPCA) provides a concise set of basis vectors which capture non-linear structures within large data sets, and is a central tool in data analysis and learning. To allow for non-linear relations, typically a full n×n kernel matrix is constructed over n data points, but this…
TorchKM: A GPU-Oriented Library for Kernel Learning and Model Selection
problem Kernel learning and model selection
method GPU acceleration
result Competitive predictive performance with speedups
Skyformer uses Gaussian kernel and Nyström method to speed up self-attention in transformers.
problem High computational cost of self-attention in transformers.
method Replaces softmax with Gaussian kernel and applies Nyström method for matrix approximation.
result Skyformer achieves comparable or better performance with fewer computation resources.
Recent advances in matrix completion enable data imputation in full-rank matrices by exploiting low dimensional (nonlinear) latent structure. In this paper, we develop a new model for high rank matrix completion (HRMC), together with batch and online methods to fit the model and out-of-sample extension to complete new …
New method estimates large matrices' spectra from small sub-matrices.
problem Estimating large matrices' spectra when full matrix-vector products are not available.
method Free decompression based on free probability theory.
result Estimates eigenspectrum of impalpable matrices from small sub-matrices.
TKRR improves KRR performance by aligning target functions with kernels.
problem Improving kernel ridge regression performance through target alignment.
method Focuses on truncated kernel ridge regression (TKRR) with an additional spectral truncation parameter.
result TKRR can achieve faster rates than full KRR, reaching parametric rates.
Determinantal point processes (DPPs) have garnered attention as an elegant probabilistic model of set diversity. They are useful for a number of subset selection tasks, including product recommendation. DPPs are parametrized by a positive semi-definite kernel matrix. In this work we present a new method for learning th…
Accelerates GPR with localized kernels for faster performance.
problem Speeding up Gaussian process regression.
method Localization kernels applied at each data point to down-weight distant points, leading to a sparsified Gram matrix.
result Significant speedups with competitive performance compared to other methods.
Robust principal component analysis (RPCA) can recover low-rank matrices when they are corrupted by sparse noises. In practice, many matrices are, however, of high-rank and hence cannot be recovered by RPCA. We propose a novel method called robust kernel principal component analysis (RKPCA) to decompose a partially cor…
Automates GPU kernel optimization for diverse applications.
problem Lack of systematic evaluation for multi-scenario GPU kernel optimization.
method Introduces MSKernelBench and CUDAMaster for automated optimization.
result Significant speedups across various operators, outperforming existing tools.
Two methods solve kernel ridge regression problems efficiently.
problem Solving kernel ridge regression problems with large datasets.
method RPCholesky and KRILL preconditioning techniques.
result Efficient solutions to KRR problems with strong guarantees.
We investigate how to train kernel approximation methods that generalize well under a memory budget. Building on recent theoretical work, we define a measure of kernel approximation error which we find to be more predictive of the empirical generalization performance of kernel approximation methods than conventional me…
Attention tokens are group elements, negated algebra norms.
problem Attention mechanism for matrix Lie groups
method Lie-Algebra Attention
result Score matches learned kernel on invariant poses, outperforms vector-token baseline
This paper develops new methods to recover the missing entries of a high-rank or even full-rank matrix when the intrinsic dimension of the data is low compared to the ambient dimension. Specifically, we assume that the columns of a matrix are generated by polynomials acting on a low-dimensional intrinsic variable, and …
Novel algorithm speeds up log-determinant estimation for large matrices.
problem Efficiently estimating log-determinants of large positive definite matrices under memory constraints.
method Hierarchical algorithm based on block-wise computation of LDL decomposition.
result Accurate estimation of NTK log-determinants from a tiny fraction of the full dataset.
A well-recognized limitation of kernel learning is the requirement to handle a kernel matrix, whose size is quadratic in the number of training examples. Many methods have been proposed to reduce this computational cost, mostly by using a subset of the kernel matrix entries, or some form of low-rank matrix approximatio…
DSoftKI scales GP regression with full derivative observations.
problem Efficiently fitting and predicting full derivative observations in Gaussian Processes.
method Extends SoftKI by using local temperature vectors for interpolation, enabling encoding of local directional sensitivity.
result DSoftKI achieves accurate predictions and scales to larger datasets with full derivative observations.
Adaptive stochastic gradient methods such as AdaGrad have gained popularity in particular for training deep neural networks. The most commonly used and studied variant maintains a diagonal matrix approximation to second order information by accumulating past gradients which are used to tune the step size adaptively. In…
Efficient kernel methods for large datasets using GPU acceleration.
problem Handling large-scale nonparametric learning problems efficiently.
method Preconditioned gradient solver, GPU acceleration, parallelization, out-of-core linear algebra, numerical precision optimization.
result Dramatic speedups on datasets with billions of points, maintaining state-of-the-art performance.
Adaptive regularization methods pre-multiply a descent direction by a preconditioning matrix. Due to the large number of parameters of machine learning problems, full-matrix preconditioning methods are prohibitively expensive. We show how to modify full-matrix adaptive regularization in order to make it practical and e…
Accelerated RPCholesky speeds up kernel matrix approximations.
problem Efficiently approximating large kernel matrices.
method Accelerated randomly pivoted Cholesky (RPCholesky) with block matrix computations and rejection sampling.
result Approximates kernel matrices up to 40 times faster.
Kernel ridge regression (KRR) is a well-known and popular nonparametric regression approach with many desirable properties, including minimax rate-optimality in estimating functions that belong to common reproducing kernel Hilbert spaces (RKHS). The approach, however, is computationally intensive for large data sets, d…
Paper presents a new way to estimate model changes without full model evaluation.
problem Efficiently estimating changes in model parameters and outputs due to data point removal.
method Dual representation of influence functions for linearizable models, reducing computational complexity.
result The dual representation can be an efficient alternative to original influence functions, especially for large models.
Constructing the adjacency graph is fundamental to graph-based clustering. Graph learning in kernel space has shown impressive performance on a number of benchmark data sets. However, its performance is largely determined by the chosen kernel matrix. To address this issue, the previous multiple kernel learning algorith…
Kernel-based clustering algorithm can identify and capture the non-linear structure in datasets, and thereby it can achieve better performance than linear clustering. However, computing and storing the entire kernel matrix occupy so large memory that it is difficult for kernel-based clustering to deal with large-scale …
Transformers with linear space and time complexity for accurate attention estimation.
problem Efficiently estimating attention in large-scale tasks without relying on priors.
method Performers use Fast Attention Via positive Orthogonal Random features (FAVOR+) for linear approximation of softmax attention.
result Performers achieve competitive results on various tasks, demonstrating the effectiveness of their attention-learning approach.
Determinantal point processes (DPPs) are an elegant model for encoding probabilities over subsets, such as shopping baskets, of a ground set, such as an item catalog. They are useful for a number of machine learning tasks, including product recommendation. DPPs are parametrized by a positive semi-definite kernel matrix…
We extend kernelized matrix factorization with a fully Bayesian treatment and with an ability to work with multiple side information sources expressed as different kernels. Kernel functions have been introduced to matrix factorization to integrate side information about the rows and columns (e.g., objects and users in …
Paper presents a fast and adaptive filter for SI suppression in full-duplex transceivers.
problem Self-interference suppression in full-duplex transceivers with nonlinearity.
method Adaptive projected subgradient method (APSM) in a reproducing kernel Hilbert space (RKHS).
result The proposed method achieves favorable digital SIC performance compared to benchmarks.
Exact and scalable algorithm for Gaussian process regression with Matérn correlations.
problem Efficient Gaussian process regression with Matérn correlations.
method Novel kernel packet theory and sparse representation of covariance matrix.
result Significantly superior to existing alternatives in computational time and predictive accuracy.
We present in this work a new family of kernels to compare positive measures on arbitrary spaces $\Xcal$ endowed with a positive kernel κ, which translates naturally into kernels between histograms or clouds of points. We first cover the case where $\Xcal$ is Euclidian, and focus on kernels which take into account th…
Algorithm POLO learns low-rank MDPs with adversarial changes in full-info feedback.
problem Learning low-rank MDPs with adversarial changes and unknown transition probabilities.
method Policy optimization-based algorithm POLO with regret guarantee.
result POLO achieves sublinear regret guarantee with no dependence on state space size.
Recent studies utilize multiple kernel learning to deal with incomplete-data problem. In this study, we introduce new methods that do not only complete multiple incomplete kernel matrices simultaneously, but also allow control of the flexibility of the model by parameterizing the model matrix. By imposing restrictions …
Efficient optimization method reduces Full AdaGrad complexity.
problem Computational challenges in Full AdaGrad for large-scale applications.
method Recursive method and streaming variant for parameter updates.
result Significant reduction in complexity and resource demands.
Identifies conditions for multiple invariant probabilities in Markov kernels.
problem Global irreducibility and recurrence do not guarantee uniqueness of invariant probabilities.
method Uses Jordan decomposition of the difference of two invariant probabilities.
result A Markov kernel has more than one invariant probability if and only if it admits a visible absorbing decomposition.
Study shows deterministic equivalent for neural network kernel convergence.
problem Understanding convergence of neural network kernels.
method Analyzes empirical spectral distribution of Conjugate Kernel, proving convergence to a deterministic limit.
result Obtains a deterministic equivalent for the Stieltjes transform and resolvent of the Conjugate Kernel.
SKI accelerates GP inference with sparse grids to handle higher dimensions.
problem SKI scales poorly in high dimensions due to dense grid size.
method Sparse grids within SKI framework, novel matrix-vector multiplication algorithm.
result SKI can be scaled to higher dimensions while maintaining accuracy.
New algorithm tackles low-rank constraints in optimal transport problems.
problem Optimal transport problems with low-rank constraints.
method Explicit factorization of low-rank couplings as a product of sub-coupling factors linked by a common marginal.
result Stationary convergence of the algorithm proved.
New GPU kernels boost deep learning speed and memory efficiency.
problem Sparse deep learning matrices are not well-suited for existing sparse kernels.
method Identified favorable properties of sparse matrices from deep learning, developed high-performance GPU kernels for sparse matrix operations.
result 27% of single-precision peak performance on Nvidia V100 GPUs achieved with new kernels.
Most kernel-based methods, such as kernel or Gaussian process regression, kernel PCA, ICA, or k-means clustering, do not scale to large datasets, because constructing and storing the kernel matrix Kn requires at least O(n2) time and space for n samples. Recent works show that sampling point…
Tensor network surrogate for efficient option pricing in large portfolios.
problem Large-scale portfolio revaluation problems in market risk management.
method Tensor-train (TT) approximation for high-dimensional price surfaces, direct inference using Laplacian kernel and TT representations.
result Tensor surrogate achieves lower test error and faster evaluation times compared to standard GPR.