Novel framework for policy optimization with general parameterization and linear convergence.
problem Lack of theoretical guarantees for policy optimization with general parameterization schemes.
method Mirror descent approach for policy optimization with general parameterization.
result First result of linear convergence for policy-gradient-based method with general parameterization.
Stochastic parameterizations account for uncertainty in the representation of unresolved sub-grid processes by sampling from the distribution of possible sub-grid forcings. Some existing stochastic parameterizations utilize data-driven approaches to characterize uncertainty, but these approaches require significant str…
New parameterization for (1,1)-knots simplifies their study.
problem Parameterizing (1,1)-knots in a novel way. method From minimal-length representatives of arcs in the multipunctured plane.
result Introduced parameterization is essentially unique for satellite (1,1)-knots. Empirical studies show that gradient-based methods can learn deep neural networks (DNNs) with very good generalization performance in the over-parameterization regime, where DNNs can easily fit a random labeling of the training data. Very recently, a line of work explains in theory that with over-parameterization and p…
Improves adaptivity in sequence models by over-parameterizing.
problem Adaptivity and generalization in sequence models.
method Over-parameterized gradient descent using eigenfunctions.
result Over-parameterization enhances model adaptivity and generalization.
Small parameterized towers improve multi-task learning efficiency and generalization.
problem Balancing Pareto efficiency and generalization in multi-task learning.
method Under-parameterized self-auxiliaries for multi-task models.
result Small parameterized towers enhance Pareto efficiency in various multi-task applications.
In this paper, we introduce a parameterized discrete curvature (α-curvature) for piecewise linear metrics on polyhedral surfaces, which is a generalization of the classical discrete curvature. A discrete uniformization theorem is established for the parameterized discrete curvature, which generalizes the discrete uni…
Least squares regression shows unexpected double descent in under-parameterized models.
problem Understanding the generalization of under-parameterized models in regression.
method Analyzing the spectrum and eigenvectors of the sample covariance matrix.
result Least squares regression can exhibit a peak in generalization in the under-parameterized regime, contrary to previous explanations.
This paper explores adaptive methods in over-parameterized linear regression.
problem Understanding why neural networks generalize well in over-parameterized settings.
method Characterizes two sub-classes of adaptive methods and their generalization performance.
result Adaptive methods in over-parameterized linear regression converge to the minimum norm solution.
The current paper discusses some new results about conformal polynomic surface parameterizations. A new theorem is proved: Given a conformal polynomic surface parameterization of any degree it must be harmonic on each component. As a first geometrical application, every surface that admits a conformal polynomic paramet…
The paper shows how the generalization curve can have multiple peaks, influenced by data and learning algorithm biases.
problem Understanding the generalization behavior of linear regression models under varying parameterizations.
method Analyzes generalization loss in linear regression models with varying parameterizations, both under- and over-parameterized.
result The generalization curve can have an arbitrary number of peaks, and their locations can be controlled.
In the overview, a generic mathematical object (mapping) is introduced, and its relation to model physics parameterization is explained. Machine learning (ML) tools that can be used to emulate and/or approximate mappings are introduced. Applications of ML to emulate existing parameterizations, to develop new parameteri…
We introduce a new parameterization method for deep learning layers using spectral tensor train decomposition.
problem Efficiency and stability in deep learning models with weight matrix compression.
method Spectral Tensor Train Parameterization (STTP) of weight matrices.
result Improved compression and training stability in neural networks.
Novel approach analyzes ReLU networks' training dynamics and proposes GmP for improved optimization.
problem Stochastic optimization instability in ReLU networks impedes convergence and generalization.
method Characteristic activation boundaries analysis and Geometric Parameterization (GmP) technique.
result GmP resolves instability, leading to better optimization, convergence, and generalization.
Gradient EM converges globally for over-parameterized Gaussian mixtures.
problem Global convergence of gradient EM for Gaussian mixtures with more than 2 components.
method Likelihood-based convergence analysis framework.
result Gradient EM converges globally with a sublinear rate of O(1/√t).
Study bounds graph neural networks' over-parameterized error.
problem Understanding graph neural networks' performance in over-parameterized regimes.
method Developed mean-field regime bounds for graph convolutional and message passing neural networks.
result Established upper bounds with a convergence rate of O(1/n) for generalization error. Study shows how over-parameterized classifiers can still perform well on noisy data.
problem Understanding how maximum margin classifiers perform in over-parameterized settings with noisy data.
method Analyzes maximum margin classifiers on sub-Gaussian mixtures, providing risk bounds.
result Characterizes conditions for 'benign overfitting' in linear classification problems.
This paper introduces a new bound to explain generalization in over-parameterized models.
problem Understanding why some over-parameterized models generalize well while others do not.
method PAC-Chernoff bounds and smoothness measures based on large deviation theory.
result Interpolators with smoother structures generalize better, according to the new theoretical framework.
The paper introduces various canonical parameterizations for 2D-curved shapes.
problem Comparing unparameterized simple curves in the plane.
method Proposes diverse canonical parameterizations, including arc-length and curvature-based.
result Natural parameterizations correspond to physical movements and are geometric invariants.
Point cloud is the most fundamental representation of 3D geometric objects. Analyzing and processing point cloud surfaces is important in computer graphics and computer vision. However, most of the existing algorithms for surface analysis require connectivity information. Therefore, it is desirable to develop a mesh st…
Over-parameterized models can memorize noisy labels and still generalize well, revealing a hidden structure.
problem Understanding how over-parameterized models can simultaneously memorize noisy labels and generalize well.
method Investigated through modular arithmetic tasks with label noise using two-layer neural networks.
result Over-parameterized models can achieve near-perfect test accuracy with 80% label noise by extracting an internal generalization structure.
The paper analyzes optimal implicit bias in linear regression for over-parameterized models.
problem Finding the best generalization performance in over-parameterized linear regression.
method Asymptotic analysis of generalization performance for convex functions/potentials.
result Optimal implicit bias that achieves the best generalization error under certain conditions.
New insights into optimization and generalization for linear models.
problem Understanding the implicit regularization of optimization methods for linear models.
method Investigating the norms minimized by interpolating solutions and using projections to move between solutions.
result Proving that for over-parameterized linear classification, projections onto the data-span enable the use of under-parameterized techniques.
There are currently two parameterizations used to derive fixed kernels corresponding to infinite width neural networks, the NTK (Neural Tangent Kernel) parameterization and the naive standard parameterization. However, the extrapolation of both of these parameterizations to infinite width is problematic. The standard p…
MO-PaDGAN improves multi-objective optimization by generating diverse and high-performing designs.
problem Challenges in parameterizing engineering designs for multi-objective optimization.
method MO-PaDGAN uses a generative adversarial network with a Determinantal Point Processes loss function to address these challenges.
result MO-PaDGAN generates designs with improved performance and coverage, even surpassing training data.
Over-parameterized models reduce Out-of-Distribution (OOD) generalization loss.
problem Understanding how over-parameterized models handle non-trivial distributional shifts.
method Investigating random feature models and examining non-trivial natural distributional shifts.
result Increasing model parameterization reduces OOD loss.
The paper explores how over-parameterized linear regression models generalize without violating learning theory principles.
problem Understanding how over-parameterized linear regression models generalize without violating learning theory principles.
method The paper uses the predictive normalized maximum likelihood (pNML) learner to investigate the minimum norm solution of over-parameterized linear regression models.
result The model generalizes well when the test sample lies in a subspace spanned by eigenvectors associated with large eigenvalues of the training data.
New method improves deep learning models robustness to label noise.
problem Improving deep learning models' robustness to corrupted labels.
method Sparse over-parameterization and implicit regularization.
result State-of-the-art test accuracy against label noise on various datasets.
In this paper we propose a hybrid architecture of actor-critic algorithms for reinforcement learning in parameterized action space, which consists of multiple parallel sub-actor networks to decompose the structured action space into simpler action spaces along with a critic network to guide the training of all sub-acto…
Local convergence theory for mildly over-parameterized neural nets.
problem Understanding why over-parameterization works in neural networks.
method Developed a local convergence theory for two-layer neural nets, showing neuron convergence under certain conditions.
result All student neurons converge to one of teacher neurons when the loss is below a threshold.
Kirchhoff energy is a classical functional on the space of arclength-parameterized framed curves whose critical points approximate configurations of springy elastic rods. We introduce a generalized functional on the space of framed curves of arbitrary parameterization, which model rods with axial stretch or cross-secti…
New methods solve tensor-on-tensor regression with unknown rank, revealing benefits of over-parameterization.
problem Connecting tensor responses to tensor covariates with unknown intrinsic rank.
method Riemannian gradient descent and Riemannian Gauss-Newton methods for tensor-on-tensor regression.
result Riemannian optimization methods converge linearly and quadratically to a statistically optimal estimate in rank over-parameterized settings.
Under-parameterization hinders deep RL's efficiency.
problem Implicit under-parameterization impairs data-efficiency in deep RL.
method Characterized and mitigated the rank collapse of value network features.
result Controlling rank collapse improves deep RL performance.
This work proposes a mathematical framework for loss landscapes and optimization in deep neural networks.
problem The effectiveness of gradient-based optimization in over-parameterized neural networks.
method A modern view and mathematical framework of loss landscapes and efficient optimization in over-parameterized machine learning models.
result Wide neural networks satisfy the PL∗ condition, explaining (S)GD convergence to a global minimum. CNNs improve generalization to unseen audio devices with increased width, not depth.
problem CNNs are sensitive to specific audio recording devices in acoustic scene classification.
method Investigated the relationship between over-parameterization and generalization in CNNs for audio classification.
result Increasing width improves generalization to unseen devices without increasing the number of parameters.
We propose a new sample-efficient methodology, called Supervised Policy Update (SPU), for deep reinforcement learning. Starting with data generated by the current policy, SPU formulates and solves a constrained optimization problem in the non-parameterized proximal policy space. Using supervised regression, it then con…
OPT framework improves neural network generalization by learning an orthogonal transformation.
problem Improving neural network generalization.
method Orthogonal over-parameterized training (OPT) framework that minimizes hyperspherical energy.
result OPT framework provably minimizes hyperspherical energy and improves empirical generalization.
New insights into bias and variance in over-parameterized models.
problem Understanding bias and variance in over-parameterized models.
method Analytic expressions derived from statistical physics for two minimal models.
result Over-parameterized models can overfit even in noiseless conditions.
Polynomially parameterizes knots and spheres, proving analogous results.
problem Parameterizing knots and spheres using polynomials.
method Analogous to classical knots, parameterized long 2-knots and certain classes of knotted spheres.
result Polynomial parameterizations for knotted spheres constructed.
Conformal surface parameterization is useful in graphics, imaging and visualization, with applications to texture mapping, atlas construction, registration, remeshing and so on. With the increasing capability in scanning and storing data, dense 3D surface meshes are common nowadays. While meshes with higher resolution …
Pruning improves model generalization in over-parameterized models, contradicting traditional theories.
problem Pruning's effect on generalization in over-parameterized models.
method Empirical study on standard pruning algorithms and additional regularization effects.
result Pruning leads to better training and regularization, improving generalization.
Two novel algorithms for conformal parameterization of multiply-connected surfaces.
problem Parameterization of surfaces with holes.
method Developed efficient methods using quasi-conformal theory.
result Efficient conformal parameterization of multiply-connected surfaces.
We prove generic regularity and Uhlenbeck-type compactification theorems for the moduli spaces of PU(2)-monopoles. Generic regularity is NOT obtained in the usual way (by applying Sard theorem to a smooth parameterized moduli space), since the parameterized moduli space can be a priori singular. We explain why, using t…
A recent line of research on deep learning focuses on the extremely over-parameterized setting, and shows that when the network width is larger than a high degree polynomial of the training sample size n and the inverse of the target error ε−1, deep neural networks learned by (stochastic) gradient descent enjoy …
The paper analyzes how over-parameterization affects GD convergence in matrix sensing problems.
problem Matrix sensing problem with over-parameterized gradient descent.
method Analyzes symmetric and asymmetric parameterizations, provides lower bounds and convergence rates.
result Over-parameterization slows down GD convergence, but asymmetric parameterization can speed up convergence.
We consider reinforcement learning in parameterized Markov Decision Processes (MDPs), where the parameterization may induce correlation across transition probabilities or rewards. Consequently, observing a particular state transition might yield useful information about other, unobserved, parts of the MDP. We present a…
Over-parameterization makes optimization easier for simple neural networks, even with minor extra neurons.
problem Understanding the impact of over-parameterization on optimization landscapes of shallow neural networks.
method Analyzing a simple ReLU neural network with Gaussian inputs, focusing on optimization properties and landscape changes.
result Over-parameterization makes the objective function one-point strongly convex in most directions, aiding optimization.
New method to recover over-parameterized models corrupted during estimation.
problem Recovering statistical models corrupted after initial estimation.
method Robust estimation using over-parameterized models and redundancy.
result Stochastic gradient descent is well-suited for model repair, but sparsity is generally not repairable.