KOOW method provides optimal covariate balance for continuous treatments.
problem Estimating effects of continuous treatments with robustness to model misspecification and extreme weights.
method Kernel Optimal Orthogonality Weighting (KOOW) using convex optimization.
result KOOW provides optimal covariate balance and controls for extreme weights.
Regularization techniques are widely used to improve the generality, robustness, and efficiency of deep convolutional neural networks (DCNNs). In this paper, we propose a novel approach of regulating DCNN convolutional kernels by a structured filter bank. Comparing with the existing regularization methods, such as $\el…
Orthogonal initialization does not speed up training in ultra-wide neural networks.
problem Exploring the effect of orthogonal initialization on training speed in deep neural networks.
method Study of neural tangent kernel dynamics in FCNs and CNNs with orthogonal initialization.
result The NTK of orthogonally-initialized networks remains constant during training, suggesting no speedup in the NTK regime.
Deep networks with orthogonal weights show stable fluctuations, improving generalization and training speed.
problem Fluctuations in deep networks with Gaussian weights can impair training, especially in networks with depth comparable to width.
method Analytical and numerical studies of fully-connected networks with orthogonal weight initialization and tanh activations.
result Rectangular networks with orthogonal weights have stable fluctuations independent of network depth, leading to better generalization and training speed.
A new method for approximating softmax and Gaussian kernels with reduced error.
problem Approximating softmax and Gaussian kernels with low error.
method Simplex Random Features (SimRFs) and SimRFs+.
result SimRFs provide the smallest MSE among weight-independent geometrically-coupled PRF mechanisms.
Pion optimizes LLMs by preserving weight matrix singular values.
problem Training large language models (LLMs) with standard optimizers leads to unstable weight matrices.
method Pion uses orthogonal transformations to update weight matrices, preserving their singular values.
result Pion offers a stable alternative to standard optimizers for LLM pretraining and finetuning.
Orthogonal random features approximate a Bessel kernel, offering sharper bounds than random Fourier features.
problem Approximating Gaussian kernel efficiently for large datasets.
method Use of Haar orthogonal matrices to construct orthogonal random features and analyze their bias and variance.
result Orthogonal random features approximate a Bessel kernel, not the Gaussian kernel, with sharper bounds.
Introduces new curvature concept for Kähler manifolds.
problem Optimizing curvature constraints for projective Kähler manifolds.
method Introduces weighted orthogonal Ricci curvature and proves vanishing theorems.
result Proves optimal curvature constraints for projective Kähler manifolds.
The paper connects Riemannian Gaussian distributions to random matrix theory and diffusion kernels.
problem Analyzing Riemannian Gaussian distributions on symmetric spaces.
method Analytical computation of marginals using orthogonal and skew orthogonal polynomials, and diffusion kernels.
result Riemannian Gaussian distributions are random matrix types, and their probability density functions can be computed analytically.
A method for interpreting SVMs using polynomial kernels, revealing model complexity.
problem Interpreting SVMs built with truncated orthogonal polynomial kernels.
method Orthogonal Representation Contribution Analysis (ORCA) with normalized Orthogonal Kernel Contribution (OKC) indices.
result The method reveals structural aspects of model complexity not captured by predictive accuracy.
We consider Markov Decision Problems defined over continuous state and action spaces, where an autonomous agent seeks to learn a map from its states to actions so as to maximize its long-term discounted accumulation of rewards. We address this problem by considering Bellman's optimality equation defined over action-val…
Recently mean field theory has been successfully used to analyze properties of wide, random neural networks. It gave rise to a prescriptive theory for initializing feed-forward neural networks with orthogonal weights, which ensures that both the forward propagated activations and the backpropagated gradients are near $…
A new method for disentangled representations without supervision.
problem Learning disentangled representations in unsupervised learning.
method Constr-DRKM, a deep kernel method with orthogonality constraints.
result Constr-DRKM performs similarly to β-VAE on disentanglement metrics.
We present an intriguing discovery related to Random Fourier Features: in Gaussian kernel approximation, replacing the random Gaussian matrix by a properly scaled random orthogonal matrix significantly decreases kernel approximation error. We call this technique Orthogonal Random Features (ORF), and provide theoretical…
We introduce a novel approach to perform first-order optimization with orthogonal and unitary constraints. This approach is based on a parametrization stemming from Lie group theory through the exponential map. The parametrization transforms the constrained optimization problem into an unconstrained one over a Euclidea…
We consider multi-agent stochastic optimization problems over reproducing kernel Hilbert spaces (RKHS). In this setting, a network of interconnected agents aims to learn decision functions, i.e., nonlinear statistical models, that are optimal in terms of a global convex functional that aggregates data across the networ…
One reflection suffices for orthogonal weights, reducing GPU usage.
problem Efficiently computing orthogonal weight matrices without high GPU utilization.
method Use an auxiliary neural network to compute one reflection instead of many.
result One reflection is sufficient for orthogonal weights, improving GPU utilization.
The paper defines functions that induce bounded composition operators on RKHSs with analytic positive definite functions.
problem Characterizing functions that induce bounded composition operators on RKHSs.
method Intrinsic properties of RKHSs and asymptotic properties of orthogonal polynomials.
result Only affine transforms can induce bounded composition operators in a large class of RKHSs.
Study asymptotics of extension and orthogonal Bergman kernels for high tensor powers of positive line bundles.
problem Asymptotic behavior of Bergman kernels for high tensor powers of positive line bundles.
method Analyzing the Schwartz kernel of the Ohsawa-Takegoshi extension operator and orthogonal Bergman projector, proving exponential estimates and asymptotic expansions.
result Explicit asymptotic expansions for the Ohsawa-Takegoshi extension operator and orthogonal Bergman projector.
Two new algorithms reduce online kernel regression's computational cost while maintaining optimal regret bounds.
problem Trade-off between regret and computational cost in online kernel regression.
method AOGD-ALD and NONS-ALD algorithms dynamically maintain nearly orthogonal basis to approximate kernel mapping and control approximate error.
result Achieves nearly optimal regret bounds at sublinear computational complexity.
MuonEq improves training of matrix-valued parameters by rebalancing momentum before orthogonalization.
problem Training matrix-valued parameters with orthogonalized-update optimizers like Muon.
method MuonEq introduces three lightweight pre-orthogonalization equilibration schemes: two-sided row/column normalization (RC), row normalization (R), and column normalization (C).
result Row/column normalization acts as a zeroth-order surrogate for whitening and improves the geometry seen by orthogonalization.
Orthogonal initialization speeds up convergence in deep linear networks.
problem The impact of initialization on convergence speed and model performance in deep neural networks.
method Analysis of orthogonal initialization in deep linear networks, proving its superiority over Gaussian initialization.
result Orthogonal initialization speeds up convergence relative to Gaussian initialization in deep networks.
Improved kernel quadrature with convex weights using subsampling.
problem Constructing quadrature rules with small worst-case error.
method Combining spectral properties of the kernel with recombination results.
result Effective algorithms for constructing convex quadrature rules with i.i.d. samples.
We propose new positive definite kernels for permutations. First we introduce a weighted version of the Kendall kernel, which allows to weight unequally the contributions of different item pairs in the permutations depending on their ranks. Like the Kendall kernel, we show that the weighted version is invariant to rela…
Study shows RFRR's effectiveness with nearly orthogonal data in overparameterized settings.
problem Understanding the effectiveness of random feature regression with nearly orthogonal data.
method Investigates RFRR with nearly orthogonal deterministic unit-length input data vectors in the overparameterized regime.
result Shows high-probability non-asymptotic concentration results for RFRR's training, cross-validation, and generalization errors.
New scalable GP approximation using Fourier series decomposition.
problem Scalability and accuracy in Gaussian process approximations.
method Harmonic kernel decomposition (HKD) to decompose kernels orthogonally.
result Significantly outperforms standard variational methods in scalability and accuracy.
New neural architectures with multivariate nonlinearities are optimal in function space.
problem Optimality of neural architectures with multivariate nonlinearities.
method Construction of Banach spaces via k-plane transform and sparsity-promoting norm, proving representer theorem. result Neural architectures with multivariate nonlinearities are optimal in function space.
The paper analyzes and improves a deep learning optimization technique using matrix gradient orthogonality.
problem Improving deep learning training through more effective optimization methods.
method Develops a stochastic non-Euclidean trust-region gradient method for deep learning optimization.
result Proves state-of-the-art convergence results for the proposed algorithm in various scenarios.
We consider the problem of estimating a low-rank matrix from a noisy observed matrix. Previous work has shown that the optimal method depends crucially on the choice of loss function. In this paper, we use a family of weighted loss functions, which arise naturally for problems such as submatrix denoising, denoising wit…
ParK efficiently solves kernel ridge regression for large datasets.
problem Large-scale kernel ridge regression efficiency and accuracy.
method Partitioning feature space with random projections and iterative optimization.
result Provably maintains statistical accuracy with reduced space and time complexity.
A new neural network improves classification speed and robustness.
problem Slow convergence and poor performance of traditional neural networks.
method Gegenbauer Neural Network (GNN) with R-WDD for regularized weights determination.
result GNN with R-WDD achieves comparable or better generalization performance.
Optimizes sliding window approach for tracking Gaussian densities.
problem Improving tracking performance of Gaussian density estimation.
method Theoretical analysis of sliding window Gaussian Kernel Density Estimators.
result Empirical evidence shows improved tracking performance with optimal weight sequence.
Optimizes kernel density ratios for better predictions and information measures.
problem Improving accuracy of kernel density estimates for density ratios.
method Derives an optimal weight function using calculus of variations.
result Reduces bias in kernel density estimates, leading to improved prediction posteriors and information-theoretic measures.
Study eigenvalue distributions of neural kernels for linear-width networks.
problem Eigenvalue distributions of neural kernels in linear-width networks.
method Asymptotic analysis of Conjugate Kernel and Neural Tangent Kernel under random initialization and approximate orthogonality.
result Eigenvalue distributions converge to deterministic limits, described by recursive fixed-point equations.
Proposes a new method for analyzing multimodal neuroimaging data.
problem Combining interpretability and flexibility in multimodal data analysis.
method Orthogonalized kernel debiased machine learning approach.
result Established consistency and asymptotic normality of the estimated primary parameter.
In the multiple linear regression setting, we propose a general framework, termed weighted orthogonal components regression (WOCR), which encompasses many known methods as special cases, including ridge regression and principal components regression. WOCR makes use of the monotonicity inherent in orthogonal components …
Consider the problem: given the data pair (x,y) drawn from a population with f∗(x)=E[y∣x=x], specify a neural network model and run gradient flow on the weights over time until reaching any stationarity. How does ft, the function computed by the neural network…
Bayes-optimal learning of deep random networks with Gaussian weights is studied.
problem Learning a target function corresponding to a deep, extensive-width, non-linear neural network with random Gaussian weights.
method Closed-form expressions for Bayes-optimal test error, ridge regression, kernel and random features regression are computed.
result Optimally regularized ridge regression and kernel regression achieve Bayes-optimal performances, while logistic loss yields a near-optimal test error for classification.
New scalable method balances hospital profit status and heart attack outcomes.
problem Balancing covariate distributions and minimizing weight dispersion in large datasets.
method Combines kernel basis expansion and convex optimization for efficient and flexible weighting.
result For-profit hospitals use interventional cardiology similarly to other hospitals but have higher mortality and readmission rates.
WE constructs GP kernels for mixed inputs using weighted EDMs.
problem Limitation of standard GP models in handling categorical variables.
method WEGP constructs kernel function using weighted EDMs for categorical inputs.
result WEGP improves GP model accuracy in both synthetic and real-world optimization problems.
OrthoGrad improves neural calibration by constraining gradient updates orthogonally.
problem Overconfidence in neural networks, leading to poor uncertainty estimates.
method Orthogonal gradient updates to optimize for decision boundaries and reduce overconfidence.
result Significant improvements in test loss, predictive entropy, and confidence measures.
Multiple kernel learning (MKL) method is generally believed to perform better than single kernel method. However, some empirical studies show that this is not always true: the combination of multiple kernels may even yield an even worse performance than using a single kernel. There are two possible reasons for the fail…
New method optimizes model selection in high-dimensional regression models.
problem Model selection in high-dimensional misspecified regression models with covariate shift.
method Importance-weighted orthogonal greedy algorithm (IWOGA) and high-dimensional importance-weighted information criterion (HDIWIC).
result IWOGA + HDIWIC achieves optimal convergence rates in terms of prediction error.
Semi-parametric framework for nonlinear system identification
problem Nonlinear system identification
method Orthogonal Gaussian process regression
result Interpretable models from incomplete physics
OGD proves robustness to Catastrophic Forgetting in Continual Learning.
problem Catastrophic Forgetting in Continual Learning with deep neural networks.
method Theoretical framework based on Neural Tangent Kernel for OGD.
result First generalization bound for SGD and OGD in Continual Learning.
Recurrent neural networks (RNNs) have been successfully used on a wide range of sequential data problems. A well known difficulty in using RNNs is the \textit{vanishing or exploding gradient} problem. Recently, there have been several different RNN architectures that try to mitigate this issue by maintaining an orthogo…
Positive weights improve kernel quadrature's accuracy.
problem Improving kernel quadrature weights to be positive and stable.
method Using convex geometry to approximate the kernel mean embedding with positive weights.
result Positive weights lead to improved kernel quadrature bounds with Monte-Carlo-beating rates.
Sharp fractional Sobolev inequalities on closed manifolds identified.
problem Critical fractional Sobolev embedding on closed Riemannian manifolds.
method Intrinsic heat-kernel based framework, determining optimal coefficients, proving sharp inequalities.
result Sharp p-power inequality and almost sharp inequality established.