A method for interpreting SVMs using polynomial kernels, revealing model complexity.
problem Interpreting SVMs built with truncated orthogonal polynomial kernels.
method Orthogonal Representation Contribution Analysis (ORCA) with normalized Orthogonal Kernel Contribution (OKC) indices.
result The method reveals structural aspects of model complexity not captured by predictive accuracy.
Study shows RFRR's effectiveness with nearly orthogonal data in overparameterized settings.
problem Understanding the effectiveness of random feature regression with nearly orthogonal data.
method Investigates RFRR with nearly orthogonal deterministic unit-length input data vectors in the overparameterized regime.
result Shows high-probability non-asymptotic concentration results for RFRR's training, cross-validation, and generalization errors.
The paper connects Riemannian Gaussian distributions to random matrix theory and diffusion kernels.
problem Analyzing Riemannian Gaussian distributions on symmetric spaces.
method Analytical computation of marginals using orthogonal and skew orthogonal polynomials, and diffusion kernels.
result Riemannian Gaussian distributions are random matrix types, and their probability density functions can be computed analytically.
Study reveals an equivalence principle for the spectrum of random inner-product kernel matrices in polynomial scaling.
problem Understanding the spectrum of random kernel matrices in polynomial scaling regimes.
method Investigates random matrices with nonlinear kernel functions applied to inner products of uniformly distributed vectors.
result The spectrum of the random kernel matrix is asymptotically equivalent to a simpler matrix model through free additive convolution.
Orthogonal random features approximate a Bessel kernel, offering sharper bounds than random Fourier features.
problem Approximating Gaussian kernel efficiently for large datasets.
method Use of Haar orthogonal matrices to construct orthogonal random features and analyze their bias and variance.
result Orthogonal random features approximate a Bessel kernel, not the Gaussian kernel, with sharper bounds.
Statistical leverage scores emerged as a fundamental tool for matrix sketching and column sampling with applications to low rank approximation, regression, random feature learning and quadrature. Yet, the very nature of this quantity is barely understood. Borrowing ideas from the orthogonal polynomial literature, we in…
A novel online GP model captures long-term memory in sequential data.
problem Capturing long-term memory in sequential data online.
method Integrates HiPPO framework into interdomain GP, leveraging time-varying orthogonal projections as inducing variables.
result OHSVGP outperforms existing online GP methods in predictive performance, long-term memory preservation, and computational efficiency.
We study the Chern-Simons partition function of orthogonal quantum group invariants, and propose a new orthogonal Labastida-Mariño-Ooguri-Vafa conjecture as well as degree conjecture for free energy associated to the orthogonal Chern-Simons partition function. We prove the degree conjecture and some interesting cases o…
The paper studies how neural networks evolve representations, finding a unique fixed point for nonlinear activations.
problem Understanding how neural networks transform input data across layers.
method Theoretical framework for the evolution of the kernel sequence, using mean-field regime and Hermite polynomials.
result For nonlinear activations, the kernel sequence converges globally to a unique fixed point.
We present an intriguing discovery related to Random Fourier Features: in Gaussian kernel approximation, replacing the random Gaussian matrix by a properly scaled random orthogonal matrix significantly decreases kernel approximation error. We call this technique Orthogonal Random Features (ORF), and provide theoretical…
Two new algorithms reduce online kernel regression's computational cost while maintaining optimal regret bounds.
problem Trade-off between regret and computational cost in online kernel regression.
method AOGD-ALD and NONS-ALD algorithms dynamically maintain nearly orthogonal basis to approximate kernel mapping and control approximate error.
result Achieves nearly optimal regret bounds at sublinear computational complexity.
Study asymptotics of extension and orthogonal Bergman kernels for high tensor powers of positive line bundles.
problem Asymptotic behavior of Bergman kernels for high tensor powers of positive line bundles.
method Analyzing the Schwartz kernel of the Ohsawa-Takegoshi extension operator and orthogonal Bergman projector, proving exponential estimates and asymptotic expansions.
result Explicit asymptotic expansions for the Ohsawa-Takegoshi extension operator and orthogonal Bergman projector.
In this paper we study both analytic and numerical solutions of option pricing equations using systems of orthogonal polynomials. Using a Galerkin-based method, we solve the parabolic partial diferential equation for the Black-Scholes model using Hermite polynomials and for the Heston model using Hermite and Laguerre p…
Orthogonal initialization does not speed up training in ultra-wide neural networks.
problem Exploring the effect of orthogonal initialization on training speed in deep neural networks.
method Study of neural tangent kernel dynamics in FCNs and CNNs with orthogonal initialization.
result The NTK of orthogonally-initialized networks remains constant during training, suggesting no speedup in the NTK regime.
New algorithm speeds up polynomial kernel approximations.
problem Efficiently approximating polynomial kernels of high degree.
method Oblivious sketching combined with novel sampling.
result Polynomial factor slowdown removed in running time.
New scalable GP approximation using Fourier series decomposition.
problem Scalability and accuracy in Gaussian process approximations.
method Harmonic kernel decomposition (HKD) to decompose kernels orthogonally.
result Significantly outperforms standard variational methods in scalability and accuracy.
We provide new examples of diffusion operators in dimension 2 and 3 which have orthogonal polynomials as eigenvectors. Their construction rely on the finite subgroups of O(3) and their invariant polynomials.
Proposes a new method for analyzing multimodal neuroimaging data.
problem Combining interpretability and flexibility in multimodal data analysis.
method Orthogonalized kernel debiased machine learning approach.
result Established consistency and asymptotic normality of the estimated primary parameter.
A new neural network model uses polynomial chaos theory to improve neural signal processing.
problem Redundant neural signal representation in DANNs.
method Employing arbitrary polynomial chaos theory to construct orthonormal representations in DANNs.
result Improves neural signal processing by reducing redundancy and enhancing orthogonality.
Semi-parametric framework for nonlinear system identification
problem Nonlinear system identification
method Orthogonal Gaussian process regression
result Interpretable models from incomplete physics
Abundant literature has been published on approximation methods for the forward initial margin. The most popular ones being the family of regression methods. This paper describes the mathematical foundations on which these regression approximation methods lie. We introduce mathematical rigor to show that in essence, al…
OGD proves robustness to Catastrophic Forgetting in Continual Learning.
problem Catastrophic Forgetting in Continual Learning with deep neural networks.
method Theoretical framework based on Neural Tangent Kernel for OGD.
result First generalization bound for SGD and OGD in Continual Learning.
In this paper, we specify what functions induce the bounded composition operators on a reproducing kernel Hilbert space (RKHS) associated with an analytic positive definite function defined on Rd. We prove that only affine transforms can do so in a pretty large class of RKHS. Our result covers not only the …
A recent theoretical analysis shows the equivalence between non-negative matrix factorization (NMF) and spectral clustering based approach to subspace clustering. As NMF and many of its variants are essentially linear, we introduce a nonlinear NMF with explicit orthogonality and derive general kernel-based orthogonal m…
Paper improves Monte Carlo sampling with new theoretical insights and methods.
problem Improving Monte Carlo sampling for variance reduction.
method Theoretical analysis of negatively dependent random variables and novel extensions using number theory and particle algorithms.
result Near-Orthogonal Monte Carlo (NOMC) consistently outperforms Orthogonal Monte Carlo (OMC) in various applications.
In this paper we derive a series expansion for the price of a continuously sampled arithmetic Asian option in the Black-Scholes setting. The expansion is based on polynomials that are orthogonal with respect to the log-normal distribution. All terms in the series are fully explicit and no numerical integration nor any …
Study finds polynomial convergence rate for Farey sequences linked to Riemann hypothesis.
problem Understanding convergence rates of maximum mean discrepancies for Farey sequences.
method Identifying positive-semidefinite kernels and their polynomial convergence rates.
result Polynomial convergence rate of maximum mean discrepancies of Farey sequences is equivalent to the Riemann hypothesis.
Improved Gaussian process models for interpretable predictions.
problem Complex responses require high-dimensional interaction terms in additive Gaussian processes.
method Orthogonal additive kernel (OAK) with orthogonality constraint on additive functions.
result OAK models achieve similar or better predictive performance with fewer terms, retaining interpretability.
Learning the kernel functions used in kernel methods has been a vastly explored area in machine learning. It is now widely accepted that to obtain 'good' performance, learning a kernel function is the key challenge. In this work we focus on learning kernel representations for structured regression. We propose use of po…
A new method tests Expected Shortfall by analyzing both duration and severity of VaR violations.
problem Lack of separate testing for frequency and severity in ES backtesting.
method Uses bivariate orthogonal polynomials to derive moment conditions for durations and severities.
result Proposes a Wald test for identifying mis-specified components in ES models.
New theory shows EDMD works well in chaotic systems.
problem Uncertainty in EDMD's properties in chaos.
method Developed rigorous theory of EDMD on chaotic maps using OPUC and transfer operator methods.
result EDMD converges to correct limits in chaotic systems with small polynomial dictionaries.
Kernel discriminant analysis uses nonlinear embeddings to improve classification.
problem Limited effectiveness of linear discriminant analysis in capturing nonlinear features.
method Study of nonlinear embeddings in kernel discriminant analysis using polynomial and Gaussian kernels, solving generalized eigenvalue problems.
result Polynomial and Gaussian discriminants capture class differences through population moments and randomized projections.
The noncommutative A-ideal of a knot is a generalization of the A-polynomial, defined using Kauffman bracket skein modules. In this paper we show that any knot that has the same noncommutative A-ideal as the (2,2p+1)-torus knot has the same colored Jones polynomials. This is a consequence of the orthogonality relation,…
A new method for disentangled representations without supervision.
problem Learning disentangled representations in unsupervised learning.
method Constr-DRKM, a deep kernel method with orthogonality constraints.
result Constr-DRKM performs similarly to β-VAE on disentanglement metrics.
We examine a class of embeddings based on structured random matrices with orthogonal rows which can be applied in many machine learning applications including dimensionality reduction and kernel approximation. For both the Johnson-Lindenstrauss transform and the angular kernel, we show that we can select matrices yield…
OPAA estimates probability densities using functional analysis.
problem Estimating probability density functions efficiently and accurately.
method OPAA uses a parallelizable algorithm based on functional analysis to estimate probability distributions.
result OPAA provides an efficient method to estimate probability density functions and normalizing weights.
Kernel methods can learn hierarchical polynomials efficiently.
problem Learning hierarchical structure from data.
method Iteratively reweighting kernel machines using derivatives.
result Efficient learning of hierarchical polynomials.
Universal preconditioning reduces sequential prediction regret.
problem Improving sequential prediction performance.
method Convolve target sequence with orthogonal polynomial coefficients.
result First sublinear and hidden-dimension-independent regret bounds.
Many scientific questions require estimating the effects of continuous treatments. Outcome modeling and weighted regression based on the generalized propensity score are the most commonly used methods to evaluate continuous effects. However, these techniques may be sensitive to model misspecification, extreme weights o…
Paper proves convergence rates for Gaussian kernel ridge regression.
problem Understanding convergence rates for Gaussian kernel ridge regression.
method Establishes polynomial convergence rates for KRR with fixed hyperparameters.
result First polynomial convergence rates for Gaussian kernel ridge regression.
New findings on kernel regression in the quadratic regime, improving understanding of machine learning models.
problem Understanding kernel ridge regression in the quadratic asymptotic regime.
method Extended study of kernel regression to the quadratic regime, establishing approximation bounds and spectral distributions.
result Broad class of inner-product kernels exhibit behavior similar to a quadratic kernel, with precise asymptotic training and test errors characterized.
Regularization techniques are widely used to improve the generality, robustness, and efficiency of deep convolutional neural networks (DCNNs). In this paper, we propose a novel approach of regulating DCNN convolutional kernels by a structured filter bank. Comparing with the existing regularization methods, such as $\el…
Studying the isotropy orbits of compact symmetric spaces Reiswich introduced a family of explicit polynomials in one variable in order to describe the unique minimal isotropy orbit of compact symmetric spaces with Dynkin diagram of type Dm. Based on this geometric interpretation he conjectured that these polynomials…
New method combines spectral and sparse methods for Gaussian processes.
problem Efficiently fitting Gaussian processes to large datasets.
method Orthogonally decoupled variational Fourier features.
result Competitive performance on synthetic and real-world data.
We determine the expected curvature polynomial of random real projective varieties given as the zero set of independent random polynomials with Gaussian distribution, whose distribution is invariant under the action of the orthogonal group. In particular, the expected Euler characteristic of such random real projective…
Proposes FOAGP for efficient orthogonal effect decomposition of black-box computer experiments.
problem Challenges in sensitivity analysis of black-box computer experiments with complex, nonlinear functional outputs.
method Functional-output orthogonal additive Gaussian process (FOAGP) with conditional orthogonality constraint.
result Demonstrates effectiveness in orthogonal effect decomposition and variance decomposition through simulations and real-world application.
Deep Gaussian Processes with polynomial kernels can collapse rapidly without proper hyperparameter tuning.
problem The collapse of Deep Gaussian Processes with polynomial kernels without careful hyperparameter tuning.
method Analysis using the Berry-Esseen Theorem and observation of prior behavior.
result The prior of a Deep Gaussian Process collapses rapidly towards zero or places negligible mass on low norm functions without proper hyperparameter tuning.
Symplectic GP regression models Hamiltonian systems for particle tracing.
problem Efficiently modeling long-term Hamiltonian flow maps for charged particles.
method Multi-output Gaussian process regression with symplectic matrix-valued covariance function.
result Symplectic methods outperform existing approaches in learning Hamiltonian functions.