Gradient flow on softmax attention minimizes nuclear norm of weight matrices.
problem Classification with separate key and query weight matrices.
method Gradient flow on exponential loss, separability assumption, reparameterization, approximate KKT conditions.
result Gradient flow implicitly minimizes nuclear norm of weight matrices, contrasting with Frobenius norm minimization.
Estimates spectral density of large implicit matrices efficiently.
problem Estimating eigenvalues of large implicit matrices efficiently.
method Combines randomized estimation techniques to construct unbiased estimators.
result Validated methods on large-scale problems in graph theory and random matrix theory.
Factored gradient descent finds unique rank-r solution in PSD matrix sensing.
problem Finding a unique rank-r solution in over-parameterized matrix sensing.
method Factored gradient descent with PSD constraints.
result PSD constraint alone leads to a unique rank-r matrix recovery.
Mirror descent algorithm recovers low-rank matrices in matrix sensing.
problem Matrix sensing with low-rank matrices under certain conditions.
method Discrete-time mirror descent applied to empirical risk with Bregman divergence analysis.
result Mirror descent converges to a matrix minimizing a specific nuclear norm-related quantity.
Deep neural networks implicitly self-regularize, shown by random matrix theory.
problem Understanding and quantifying implicit self-regularization in deep neural networks.
method Random Matrix Theory applied to various DNNs, including pre-trained and self-trained models.
result DNN training implicitly implements self-regularization, observable in empirical spectral densities.
This paper is devoted to the characterization of differentially flat nonlinear systems in implicit representation, after elimination of the input variables, in the differential geometric framework of manifolds of jets of infinite order. We extend the notion of Lie-Bäcklund equivalence, introduced in Fliess et al. (1999…
Deep equilibrium models converge globally without explicit computation.
problem Global convergence of deep learning models with implicit layers.
method Analysis of gradient dynamics and proof of convergence rate.
result Deep equilibrium models converge to global optimum at a linear rate.
RMT reveals self-regularization in neural networks, including traditional and heavy-tailed forms.
problem Understanding and quantifying self-regularization in neural networks.
method Application of Random Matrix Theory to analyze weight matrices of various neural network models.
result Identification of 5+1 phases of training in neural networks, including traditional and heavy-tailed self-regularization.
Deep networks without non-linearities are equivalent to shallow ones.
problem Training deep orthogonal linear networks with no non-linearity.
method Riemannian gradient descent and gradient descent on factorization.
result Training deep overparametrized networks is equivalent to shallow ones.
Rank-one measurements limit feasible sets for low-rank PSD matrices.
problem Feasibility of PSD matrices under rank-one measurements.
method Characterization of feasible sets for PSD matrices given rank-one projections.
result Radius of feasible sets determines singleton solution sets for low-rank matrices.
Gradient descent recovers low-rank matrices from corrupted measurements with double over-parameterization.
problem Robust recovery of low-rank matrices from grossly corrupted measurements.
method Gradient descent with discrepant learning rates for double over-parameterized models.
result Gradient descent with discrepant learning rates provably recovers the underlying matrix without prior knowledge on rank or sparsity.
SHINE uses forward pass quasi-Newton matrices to approximate Jacobian inverses for faster bi-level optimization.
problem Efficiently solving bi-level optimization problems with large Jacobian matrices.
method Proposes using quasi-Newton matrices from the forward pass to approximate the inverse Jacobian matrix.
result Empirically shows SHINE reduces computational cost of the backward pass for various problems.
DEQs and explicit networks are nearly equivalent for Gaussian mixtures.
problem Understanding the equivalence between DEQs and explicit neural networks.
method Random matrix theory and analysis of kernel matrices.
result A shallow explicit network can mimic the kernel of a DEQ.
This paper studies geodesics between covariance matrices of different ranks using the Bures-Wasserstein metric.
problem Geodesics between covariance matrices of varying ranks.
method Analyzes the Bures-Wasserstein distance on covariance matrices, completing previous work on geodesics and providing explicit formulas.
result The set of all minimizing geodesics between two covariance matrices is parametrized by a closed unit ball in R ( k − r ) i m e s ( l − r ) \mathbb{R}^{(k-r) imes(l-r)} R ( k − r ) im es ( l − r ) . ReLU networks implicitly favor low-rank solutions, but not as strongly as linear networks.
problem Understanding implicit regularization in ReLU networks for rank minimization.
method Analysis of gradient flow on ReLU networks, empirical testing.
result Gradient flow on ReLU networks does not necessarily minimize ranks, unlike in linear networks.
Exact expressions for double descent and implicit regularization in over-parameterized models.
problem Understanding the generalization error of over-parameterized models like deep neural networks.
method Surrogate random design to replace standard i.i.d. design, leading to exact expressions for mean squared error and implicit regularization.
result Exact non-asymptotic expressions for double descent and implicit regularization in over-parameterized models.
Kernel Ridgeless Regression can generalize well without explicit regularization.
problem The challenge of achieving generalization in Kernel Ridgeless Regression without additional regularization.
method Minimum-norm interpolated solutions with high-dimensional data, curvature of kernel function, and favorable data geometry.
result Implicit regularization leads to good generalization in Kernel Ridgeless Regression.
Improved method for unbiased causal discovery in presence of unobserved confounding.
problem Unbiased data synthesis for causal discovery algorithms in the presence of unobserved confounding.
method Explicit block-hierarchical ancestral sampling to address limitations of implicit parameterization.
result Our approach fully covers the space of causal models, including those generated by implicit parameterization.
Study shows how feature weighting affects neural network regularization.
problem Understanding how feature weighting influences neural network regularization.
method Derived equivalence paths connecting different weighting matrices and ridge regularization levels.
result Ridge estimators trained on weighted features are asymptotically equivalent when evaluated against test vectors.
Paper proposes DAG-DB for learning discrete DAGs via backpropagation.
problem Learning Directed Acyclic Graphs (DAGs) from data.
method DAG-DB uses Discrete Backpropagation with I-MLE and Straight-Through Estimation.
result DAG-DB learns DAGs effectively using probabilistic sampling and backpropagation.
The paper analyzes two ISGD modes for statistical inference, deriving error bounds and confidence intervals.
problem Statistical inference with implicit SGD for smooth convex functions.
method Proximal Robbins-Monro (proxRM) and proximal Polyak-Ruppert (proxPR) procedures for ISGD.
result Derives non-asymptotic error bounds and confidence interval estimators for model parameters.
Weight Decay induces low-rank weight matrices in neural networks, improving generalization.
problem Improving generalization in neural networks.
method Training ReLU NN with Weight Decay and Stochastic Gradient Descent.
result The weight matrix of a trained NN is approximately rank-two.
The paper analyzes matrix completion with unlabeled implicit feedback and provides error bounds.
problem Matrix completion with shared low-rank ground truth and sampling distribution.
method Combining subspace recovery theory and matrix completion bounds.
result Error bounds showing contributions from estimating the sampling distribution and ground truth.
New method estimates large matrices' spectra from small sub-matrices.
problem Estimating large matrices' spectra when full matrix-vector products are not available.
method Free decompression based on free probability theory.
result Estimates eigenspectrum of impalpable matrices from small sub-matrices.
Develops a novel stochastic algorithm for diagonal estimation of large matrices.
problem Efficient diagonal estimation for large or implicit matrices.
method Adaptive parameter selection in a stochastic algorithm.
result Lower bound on random query vectors needed for estimation.
A new classification rule for FDA improves classification performance by accounting for unequal covariance matrices.
problem Unequal covariance matrices in practical situations affect the performance of FDA and its variants.
method Proposes a novel classification rule for FDA that accounts for unequal covariance matrices, applicable to many FDA variants.
result The new classification rule improves classification performance compared to original FDA and variants.
Adaptive Bayesian sampling technique simplifies mass matrix learning.
problem Complexity in learning mass matrices for adaptive samplers.
method Monte Carlo EM framework with online learning of mass matrices.
result Comparable sampling accuracy to Riemannian samplers but faster.
Efficiently optimizes CNN and RNN parameters on Stiefel manifold.
problem Computational expense in optimizing orthonormal matrices on Stiefel manifold.
method Cayley transform for efficient retraction and vector transport on Stiefel manifold.
result Cayley SGD and ADAM achieve faster convergence and less training time.
SNN architecture shows gradient descent converges to regularized solution in matrix sensing problems.
problem Understanding implicit regularization in neural networks for matrix sensing.
method Developed Spectral Neural Networks (SNN) for matrix learning problems, rigorously demonstrating implicit regularization.
result Gradient descent converges to the solution of a regularized learning problem in matrix sensing problems.
New method uses matrix powers for recommendation systems.
problem Predicting unobserved entries in sparse matrices.
method Coordinate descent algorithm to learn embeddings from higher-order matrix powers.
result Outperforms methods using only side information or second-order interactions.
Gradient descent aligns weights in deep linear networks for binary classification.
problem Aligning weights in deep linear networks for binary classification.
method Gradient descent applied to strictly decreasing loss functions.
result Normalized weight matrices align across layers, converging to the maximum margin solution.
New study shows deep networks generalize well due to loss surface geometry.
problem Why deep networks generalize well despite many parameters.
method Analyzed local geometry of loss surface and its effect on SGD.
result SGD stays close to low-dimensional subspace, leading to better generalization bounds.
Study creates web interface to elicit user-preferred metrics.
problem Eliciting classification metrics that align with user preferences.
method Developed a web-based interface and conducted a user study.
result Users preferred metrics that align with their task and context.
New method reduces memory usage for Bayesian inverse problems on large grids.
problem Solving large-scale linear inverse problems with Gaussian process priors.
method Implicit representation of posterior covariance matrices, sequential disintegrations of Gaussian measures.
result Significant reduction in uncertainty for high-density regions estimation.
Paper studies asymmetric matrix sensing, proving gradient descent converges to low-rank solutions.
problem Reconstructing asymmetric low-rank matrices from linear measurements.
method Factorized gradient descent with coupling and regularization properties.
result Gradient descent from small random initialization converges to globally optimal and generalizing solutions.
Kernel methods are an extremely popular set of techniques used for many important machine learning and data analysis applications. In addition to having good practical performances, these methods are supported by a well-developed theory. Kernel methods use an implicit mapping of the input data into a high dimensional f…
Gradient descent recovers low-rank matrices from random rank-one measurements.
problem Recovering low-rank matrices from random rank-one measurements.
method Directly estimate the low-rank factor by minimizing a nonconvex quadratic loss function via vanilla gradient descent with tailored spectral initialization.
result The algorithm converges to the ground truth with near-optimal sample and computational complexity when the true rank is small.
MLP-Mixer achieves better performance through sparsity and wider architecture.
problem Understanding why MLP-Mixer outperforms conventional MLPs.
method Revealed sparseness as a key mechanism, showed effective expression as wider MLP, demonstrated quantitative similarities, and applied a guiding principle.
result MLP-Mixer's performance improvement through sparseness and wider architecture.
Theoretical justification for deep networks' performance with regularization techniques.
problem Understanding the performance of deep networks trained with the square loss.
method Analysis of gradient flow and theoretical justification of regularization techniques.
result Convergence to solutions with smaller Frobenius norms leads to better classification error bounds.
Paper presents new matrix formats for deep neural networks that improve inference efficiency.
problem High computational cost of dot product operations in deep neural networks.
method Develops new matrix formats with bounded complexity by entropy of weight matrices.
result Up to x90 energy savings and x5 speed ups in dot product operations.
A new method constrains PARAFAC2 for better pattern recovery.
problem Challenges in analyzing multi-way measurements with variations across one mode.
method AO-ADMM approach to fit PARAFAC2 model with flexible constraints.
result The proposed method allows for flexible constraints, recovers patterns accurately, and is computationally efficient.
New algorithms optimize matrix manifolds, converging faster than existing methods.
problem Optimizing on Riemannian matrix manifolds with constraints.
method Adaptive stochastic gradient algorithms for row and column subspaces.
result Converges faster with rate O ( log ( T ) / T ) \mathcal{O}(\log (T)/\sqrt{T}) O ( log ( T ) / T ) . This paper shows how to train only the implicit layer of overparameterized implicit neural networks.
problem Understanding how the implicit layer contributes to the training of overparameterized implicit neural networks.
method Restricting training to only the implicit layer and analyzing the generalization error for ReLU-activated networks.
result Global convergence is guaranteed even if only the implicit layer is trained, and gradient flow with proper random initialization can achieve small generalization errors.
Power of network tests degrades when vertices are misaligned.
problem Power loss in network hypothesis testing due to vertex shuffling.
method Theoretical analysis and simulations of Frobenius norm differences in random dot product and stochastic block models.
result Shuffling vertices can significantly reduce the power of network tests.
Paper studies the theoretical equivalence between implicit and explicit neural networks in high dimensions.
problem Lack of theoretical analysis of implicit and explicit neural networks.
method Examined high-dimensional implicit neural networks and established their equivalence to explicit networks.
result Equivalence between implicit and explicit neural networks in high dimensions.
Gradient matching method estimates implicit regularization in complex deep learning systems.
problem Estimating implicit regularization in modern deep learning systems with complex modifications.
method Gradient matching methods to empirically estimate implicit regularization.
result Empirical estimation of implicit regularization in arbitrary networks, including dropout.
We simplify Bayesian filtering by framing it as optimization, making it practical for high-dimensional systems.
problem Bayesian filtering struggles in high-dimensional state spaces like neural networks.
method We frame Bayesian filtering as optimization, using gradient descent for nonlinear cases.
result Our method results in effective, robust, and scalable filters for high-dimensional systems.
Adversarial model improves implicit relation classification without explicit connectives.
problem Lack of explicit connectives makes implicit discourse relation classification challenging.
method Feature imitation framework with adversarial training.
result State-of-the-art performance on PDTB benchmark.