Improved sentiment analysis with any-gram kernels.
problem Sentiment classification in text data.
method Proposed a new any-gram kernel approach independent of tree kernels, making use of word embeddings more effectively.
result Significantly better performance in sentiment classification.
Deep kernel processes unify various models using Gram matrices and kernel functions.
problem Unified representation of various deep learning models.
method Defining deep kernel processes with progressively transformed Gram matrices and sampling from inverse Wishart distributions.
result Deep Gaussian processes, BNNs, infinite BNNs, and infinite BNNs with bottlenecks can all be written as deep kernel processes.
GRAM improves generative models without saddlepoint optimization.
problem Training deep generative models with saddlepoint optimization is challenging.
method GRAM uses a fixed kernel as an adversary instead of a critic network.
result GRAM networks match and outperform adversarial training in generative quality.
Many interesting machine learning problems are best posed by considering instances that are distributions, or sample sets drawn from distributions. Previous work devoted to machine learning tasks with distributional inputs has done so through pairwise kernel evaluations between pdfs (or sample sets). While such an appr…
Kernel methods are ubiquitous tools in machine learning. However, there is often little reason for the common practice of selecting a kernel a priori. Even if a universal approximating kernel is selected, the quality of the finite sample estimator may be greatly affected by the choice of kernel. Furthermore, when direc…
New kernel-based models improve on traditional neural methods in sequence modeling.
problem Sequence modeling challenges in natural language processing and neuroscience.
method Kernel-based recurrent neural networks and convolutional neural networks.
result Kernel-based models perform on par or better than traditional neural methods.
We present in this work a new family of kernels to compare positive measures on arbitrary spaces $\Xcal$ endowed with a positive kernel κ, which translates naturally into kernels between histograms or clouds of points. We first cover the case where $\Xcal$ is Euclidian, and focus on kernels which take into account th…
Linearized attention fails to converge to NTK limit even at large widths.
problem Understanding the convergence of attention mechanisms to the kernel regime.
method Analyzes linearized attention and its relationship to the NTK limit, considering practical widths and conditions.
result Linearized attention does not converge to its NTK limit at any practical width, revealing a fundamental trade-off.
Adaptive kernel model for multi-view learning reduces overfitting and scales with data.
problem Overfitting and high computational cost in multi-view learning.
method Bayesian framework, Dirichlet process Gaussian mixtures, random Fourier features, max-margin constraints, MCMC sampler.
result Adaptive learning of shift-invariant kernels from data.
New walk extraction strategies improve node embeddings in KGs.
problem Improving node embeddings in knowledge graphs.
method Proposed five different walk extraction strategies to complement basic random walks.
result The n-gram strategy performs best on average for node classification tasks.
This paper analyzes error in SKI for Gaussian Processes, providing conditions for linear time inference.
problem Lack of rigorous theoretical error analysis for SKI.
method Proved error bounds for SKI Gram matrix, examined error effects, provided practical guidelines.
result Identified two dimensionality regimes for SKI's scalability-accuracy trade-offs.
We analyze the size of the dictionary constructed from online kernel sparsification, using a novel formula that expresses the expected determinant of the kernel Gram matrix in terms of the eigenvalues of the covariance operator. Using this formula, we are able to connect the cardinality of the dictionary with the eigen…
Producing overlapping schemes is a major issue in clustering. Recent proposed overlapping methods relies on the search of an optimal covering and are based on different metrics, such as Euclidean distance and I-Divergence, used to measure closeness between observations. In this paper, we propose the use of another meas…
We present NN-grams, a novel, hybrid language model integrating n-grams and neural networks (NN) for speech recognition. The model takes as input both word histories as well as n-gram counts. Thus, it combines the memorization capacity and scalability of an n-gram model with the generalization ability of neural network…
Improved deep kernel machine achieves 94.5% test accuracy on CIFAR-10.
problem Achieving high accuracy on complex tasks like image classification.
method Stochastic kernel regularisation to add noise to learned Gram matrices.
result 94.5% test accuracy on CIFAR-10.
Generalized Gram determinant for 3-manifold invariants.
problem Invariants of 3-manifolds and bilinear forms evaluation.
method Evaluation of a bilinear form in the annulus for non-intersecting connections in the disc.
result Closed formula for the generalized Gram determinant.
Accelerates GPR with localized kernels for faster performance.
problem Speeding up Gaussian process regression.
method Localization kernels applied at each data point to down-weight distant points, leading to a sparsified Gram matrix.
result Significant speedups with competitive performance compared to other methods.
Estimates latent norms and Gram matrices for graphs on Euclidean balls.
problem Estimating latent points and their relationships in graphs on Euclidean balls.
method Estimates latent norms and Gram matrices using observed graph data.
result Graphs on Euclidean balls can have power-law degree distributions.
Kernel matrices (e.g. Gram or similarity matrices) are essential for many state-of-the-art approaches to classification, clustering, and dimensionality reduction. For large datasets, the cost of forming and factoring such kernel matrices becomes intractable. To address this challenge, we introduce a new adaptive sampli…
Paper extends KPCA using dualization for faster, more robust algorithms.
problem Efficiently perform KPCA with robustness and sparsity.
method Dualization of convex functions for multiple objective functions, promoting sparsity and robustness.
result Significant speedup in KPCA training time and improved robustness and sparsity.
New bounds on KPCA efficiency reveal conditions for fast convergence.
problem Lack of theoretical understanding of KPCA efficiency.
method Lower and upper bounds on KPCA efficiency involving empirical eigenvalues and new variance quantities.
result Fast convergence rates achievable for certain kernels, highlighting dataset properties.
Gaussian sketching preserves kernel inner products in low dimensions.
problem Preserving kernel inner products in low-dimensional spaces.
method Gaussian sketching of kernel Gram matrices and random projections in RKHS.
result Sketching yields a random projection operator that preserves weighted RKHS inner products.
Most machine learning algorithms, such as classification or regression, treat the individual data point as the object of interest. Here we consider extending machine learning algorithms to operate on groups of data points. We suggest treating a group of data points as an i.i.d. sample set from an underlying feature dis…
Paper analyzes why LISTA accelerates sparse coding.
problem Sparse coding optimization with generic methods.
method Matrix factorization of Gram kernel for acceleration.
result Improved convergence for LISTA compared to non-adaptive methods.
A new neural network model extends word embedding vectors with MeSH concepts for biomedical semantic similarity.
problem Eliciting semantic similarity between biomedical concepts remains challenging.
method Proposes a MeSH-gram neural network model that extends skip-gram by using MeSH descriptors.
result MeSH-gram outperforms skip-gram and is comparable to best methods but requires more computation and external resources.
The article derives a novel Gram-Charlier A (GCA) Series based Extended Rule-of-Thumb (ExROT) for bandwidth selection in Kernel Density Estimation (KDE). There are existing various bandwidth selection rules achieving minimization of the Asymptotic Mean Integrated Square Error (AMISE) between the estimated probability d…
We investigate the Gram determinant of the bilinear form based on curves in a planar surface, with a focus on the disk with two holes. We prove that the determinant based on n−1 curves divides the determinant based on n curves. Motivated by the work on Gram determinants based on curves in a disk and curves in an an…
A new Gaussian process framework uses neural feature maps for scalable, accurate inference.
problem Efficient and accurate Gaussian process inference for diverse data types.
method Neural feature maps to construct expressive kernels, with theoretical guarantees and practical scalability.
result The approach outperforms existing methods in accuracy and efficiency across various data modalities.
SGD benefits from a directional bias in kernel regression models.
problem Improving generalization in kernel regression models.
method Generalized directional bias property of SGD in kernel regression.
result SGD converges along the eigenvector of the largest eigenvalue of the Gram matrix.
Neural-Kernel CME tackles scalability and expressiveness challenges in conditional distribution representation.
problem Scalability and expressiveness challenges in kernel conditional mean embeddings.
method Combines deep learning with CMEs using a neural network optimization framework.
result Achieves competitive and often superior performance in conditional density estimation and RL.
Study Gram determinants in knot theory, focusing on a Möbius band determinant.
problem Closed formula for the Gram determinant of type (Mb)1. method Survey of Gram determinants, focusing on a Möbius band determinant.
result Speculation on closed formula for (Mb)1 Gram determinant. KiloGrams finds top-k large n-grams for malware classification.
problem Lack of efficient methods for large n-grams in malware classification.
method Developed a fast method for finding top-k large n-grams.
result Large n-grams improve malware classification and provide interpretable features.
The paper connects Chebyshev polynomials and Gram determinants on Möbius bands.
problem Exploring the relationship between Chebyshev polynomials and Gram determinants on Möbius bands.
method Analyzing Mersenne numbers and Chebyshev polynomials, proving conjectures, and developing algorithms.
result A factor of the Gram determinant supports a conjecture about its closed formula involving Chebyshev polynomials.
We propose a new class of convex penalty functions, called \emph{variational Gram functions} (VGFs), that can promote pairwise relations, such as orthogonality, among a set of vectors in a vector space. These functions can serve as regularizers in convex optimization problems arising from hierarchical classification, m…
A new GGN method speeds up training of deep neural networks for regression tasks.
problem Training deep neural networks efficiently for regression problems.
method Proposes a Gram-Gauss-Newton (GGN) algorithm for overparameterized neural networks.
result For sufficiently wide neural networks, GGN achieves quadratic convergence rate.
Word2vec skip-gram with negative sampling is explained as a weighted logistic PCA.
problem Understanding word2vec skip-gram with negative sampling.
method Equivalence to weighted logistic PCA, comparison to other methods, extension to higher dimensions.
result Word2vec skip-gram with negative sampling is a weighted logistic PCA.
GRAM generates scalable graphs with a novel attention mechanism.
problem Scalability in graph generation for large datasets.
method GRAM uses a graph attention mechanism to generate scalable graphs.
result GRAM outperforms baseline methods in scalability and quality.
Kernel and neural embeddings improve optimization and generalization in deep networks.
problem Improving optimization and generalization in deep neural networks.
method Investigated three kernel representations and their neural network approximations, comparing their optimization and generalization properties.
result Kernel and neural embeddings enhance both optimization and generalization in deep networks.
New methods extend kernel estimators for partial rankings, improving performance in machine learning tasks.
problem Incomplete rankings data in real-world applications.
method Antithetic and Monte Carlo kernel estimators for partial rankings, variance reduction scheme.
result Improved antithetic kernel estimator with lower variance and better performance.
Effective Gram matrix predicts deep network generalization.
problem Understanding and predicting deep network generalization.
method Derived a differential equation governing generalization gap, analyzed with effective Gram matrix.
result Effective Gram matrix accurately predicts test loss during training.
This paper speeds up kernel methods using sparsified Gaussian sketches.
problem Kernel methods' computational limitations.
method Sparsified Gaussian sketches for kernel methods.
result Efficient time and space savings for kernel methods.
Improved variational approximation for deep Wishart process models.
problem Improving predictive performance of deep Wishart process models.
method Generalizing the Bartlett decomposition of the Wishart distribution to allow linear combinations of rows and columns.
result Better predictive performance achieved with minimal additional computation cost.
Novel Newton method for large-scale kernel methods using random features.
problem Efficiently solving large-scale finite-sum minimization problems in RKHS.
method Randomized feature-based Newton method for empirical risk minimization.
result Local superlinear and global linear convergence of the method.
A new method for deep Wishart processes improves kernel-based models.
problem Inference in deep Wishart processes is challenging due to the need for flexible distributions over positive semi-definite matrices.
method Developed a novel approach to flexible distributions over positive semi-definite matrices using the Bartlett decomposition of the Wishart probability density. Used this to create an approximate posterior for the DWP.
result Improved performance of inference in the DWP compared to DGP with equivalent prior.
GRAM addresses healthcare data insufficiency and interpretation challenges using graph-based attention.
problem Data insufficiency and lack of interpretability in healthcare predictive modeling.
method GRAM integrates EHR with medical ontologies, using attention mechanisms to represent medical concepts.
result GRAM outperforms RNN in accuracy and interpretability, using less data.
New Gram determinant from Möbius band connects to annulus case.
problem Exploring new Gram determinants in knot theory.
method Skein theoretic approach and bilinear forms.
result Proves important results about new Gram determinant structure.
Simple bounds for covariance and Gram matrices across various settings.
problem Capturing the behavior of smaller eigenvalues in covariance and Gram matrices.
method General-purpose theorem converting uniform bounds into relative bounds.
result Sharper control of eigenvalues across the spectrum.
Proposes a faster Transformer decoding method by truncating target-side self-attention windows.
problem Efficiency in Transformer decoding with minimal BLEU score loss.
method N-gram assumption to truncate target-side self-attention windows.
result N-gram masked self-attention model maintains BLEU score for N values from 4 to 8.