Study finds implicit government guarantee improves municipal investment bond ratings.
problem Questioning the objectivity of municipal investment bond ratings due to implicit government guarantee.
method Text mining of policy documents and PMC index model for implicit guarantee strength calculation.
result Implicit government guarantee boosts municipal investment bond ratings, especially in less developed regions.
This paper measures the intensity of implicit government guarantees using PMC index model.
problem Excessive local government debt due to implicit government guarantees.
method Text mining of policy documents related to municipal investment bonds, PMC index model.
result Recent policies have reduced the intensity of implicit government guarantees.
Study shows SGD's generalization is not explained by implicit bias.
problem Explaining the generalization ability of overparameterized learning algorithms.
method Revisited Stochastic Convex Optimization with SGD, demonstrating limitations of implicit bias.
result No distribution-independent or distribution-dependent implicit regularizer can explain SGD's generalization.
New model outperforms Neural ODEs while being more efficient.
problem Stable convergence and existence guarantees for implicit-depth models.
method Developed Monotone Operator Equilibrium Network (monDEQ) based on monotone operator theory.
result MonDEQ models outperform Neural ODEs and are more computationally efficient.
FA algorithm provides convergence guarantees for deep linear networks.
problem Training efficiency and convergence of deep neural networks.
method Theoretical analysis of Feedback Alignment (FA) algorithm for deep linear networks.
result Certain initializations lead to implicit anti-regularization, affecting learning effectiveness.
Learning algorithms for implicit generative models can optimize a variety of criteria that measure how the data distribution differs from the implicit model distribution, including the Wasserstein distance, the Energy distance, and the Maximum Mean Discrepancy criterion. A careful look at the geometries induced by thes…
The paper provides theoretical guarantees for transformation-based models in variational inference.
problem Theoretical justification for transformation-based models in variational inference.
method Theoretical analysis of non-linear latent variable models and Gaussian process priors.
result Theoretical guarantees for implicit variational inference, achieving optimal risk bounds and approximating the true posterior.
Modern implicit generative models such as generative adversarial networks (GANs) are generally known to suffer from issues such as instability, uninterpretability, and difficulty in assessing their performance. If we see these implicit models as dynamical systems, some of these issues are caused by being unable to cont…
Modern implicit generative models such as generative adversarial networks (GANs) are generally known to suffer from issues such as instability, uninterpretability, and difficulty in assessing their performance. If we see these implicit models as dynamical systems, some of these issues are caused by being unable to cont…
This paper shows how to train only the implicit layer of overparameterized implicit neural networks.
problem Understanding how the implicit layer contributes to the training of overparameterized implicit neural networks.
method Restricting training to only the implicit layer and analyzing the generalization error for ReLU-activated networks.
result Global convergence is guaranteed even if only the implicit layer is trained, and gradient flow with proper random initialization can achieve small generalization errors.
SAM improves generalization in overparameterized models, but its behavior in tensorized models is less understood.
problem Understanding the implicit regularization of SAM in tensorized models.
method Scale-invariance analysis and gradient flow analysis to derive Norm Deviation as a measure of core norm imbalance, and propose Deviation-Aware Scaling (DAS).
result DAS achieves competitive or improved performance over SAM, while offering reduced computational overhead.
Deep equilibrium models converge globally without explicit computation.
problem Global convergence of deep learning models with implicit layers.
method Analysis of gradient dynamics and proof of convergence rate.
result Deep equilibrium models converge to global optimum at a linear rate.
RLVR training dynamics reveal an implicit curriculum that shapes learning progression.
problem Understanding how RLVR overcomes the long-horizon barrier.
method Developed a theory of training dynamics for RLVR on transformers, using Fourier analysis on finite groups.
result Mixed-difficulty training naturally follows an implicit curriculum, shaping the learning progression from easy to hard.
This paper proves SGD converges to global minimum for over-parameterized ReLU networks.
problem Theoretical understanding of implicit neural networks is limited.
method Gradient flow analysis of ReLU activated implicit neural networks.
result Randomly initialized gradient descent converges to global minimum at a linear rate for square loss function in over-parameterized ReLU networks.
This work analyzes how transformers learn common linear regression tasks.
problem Understanding how in-context learning operates in real-world applications with common task structures.
method Analyzing a linear attention model trained on low-rank regression tasks.
result Statistical fluctuations in finite pre-training data induce an implicit regularization, leading to a sharp phase transition in generalization error.
Sparse Bayesian learning algorithm for estimating interaction kernels in Motsch-Tadmor model.
problem Data-driven identification of asymmetric interaction kernels in the Motsch-Tadmor model.
method Variational framework reformulating kernel identification as a subspace identification problem; sparse Bayesian learning algorithm with informative priors.
result Accurate, robust, and interpretable estimation of interaction kernels across various noise levels and data regimes.
Unified theory for semi-implicit variational inference, bridging approximation and optimization.
problem Developing a statistical theory for semi-implicit variational inference.
method Unified theory combining approximation and optimization analyses.
result Unified theory characterizes SIVI's ability to recover target distributions and governs asymptotic behavior.
Kernel semi-implicit variational inference improves variational inference without additional optimization.
problem Intractability of hierarchical semi-implicit distributions in variational inference.
method Kernel semi-implicit variational inference (KSIVI) using kernel methods to eliminate lower-level optimization.
result KSIVI reduces variational inference to kernel Stein discrepancy (KSD) optimization, improving expressiveness and tractability.
Botswana's agricultural credit scheme is unsustainable and needs reform.
problem The current scheme is unsustainable and based on flawed actuarial principles.
method Proposes a new subsidy and premium rate setting method considering non-drought years.
result A sustainable agricultural credit guarantee scheme is recommended.
Bayesian approach optimizes in-context learning for state space models.
problem Optimizing in-context learning for state space models.
method Bayesian optimal sequential prediction over latent sequence tasks.
result Bayesian optimal predictor converges to posterior predictive mean.
INO learns physical models with momentum conservation laws.
problem Learning physical models without preserving fundamental laws.
method Designing an invariant neural operator that automatically satisfies momentum conservation laws.
result The model learns complex material behaviors and achieves state-of-the-art accuracy and efficiency.
Large batch training with DP-SGD reduces model performance due to implicit bias.
problem Large batch training with DP-SGD reduces model performance.
method The study analyzes the phenomenon of implicit bias in Noisy-SGD (DP-SGD without clipping) and its theoretical solutions for linear models.
result The implicit bias in large batch training with DP-SGD is amplified by additional noise, similar to SGD.
We propose graph-dependent implicit regularisation strategies for distributed stochastic subgradient descent (Distributed SGD) for convex problems in multi-agent learning. Under the standard assumptions of convexity, Lipschitz continuity, and smoothness, we establish statistical learning rates that retain, up to logari…
Kernel SIVI improves variational inference by avoiding lower-level optimization.
problem Intractable densities in semi-implicit variational distributions.
method Kernel SIVI-SM uses a minimax formulation and kernel tricks to avoid lower-level optimization.
result Kernel Stein discrepancy (KSD) objective is computable and leads to convergence guarantees.
This paper introduces a neural sampler for scalable sampling from complex distributions.
problem Efficiently sampling from high-dimensional un-normalized distributions.
method Neural implicit sampler trained with KL and Fisher divergence methods.
result The neural sampler generates large batches of samples with low computational costs.
RLVR learning dynamics naturally create an implicit curriculum for transformers.
problem How rewards based on final outcomes help overcome the long-horizon barrier in reasoning models.
method Developed a theory of training dynamics for RLVR on transformers, using Fourier analysis on finite groups.
result Mixed-difficulty training induces an implicit curriculum that shapes the learning progression from easy to hard.
XLVINs improve data efficiency in implicit planning by leveraging latent space.
problem Improving data efficiency in implicit planning algorithms.
method XLVINs use a high-dimensional latent space to perform planning computations, breaking the algorithmic bottleneck.
result XLVINs significantly improve data efficiency across various settings compared to value iteration-based implicit planners and model-free baselines.
The paper examines how gradient descent stabilizes low-rank matrix factorization in noisy conditions.
problem Stability of low-rank implicit regularization in perturbed deep matrix factorization.
method Derives spectral conditions for gradient descent to exhibit a low-rank phase in noiseless settings and analyzes perturbed dynamics.
result Gradient descent converges to a low-rank solution under perturbation, with explicit dependence on perturbation size.
EarlyStopping package helps prevent overfitting in iterative learning procedures.
problem Preventing overfitting in iterative learning procedures.
method Early stopping implemented in a Python package.
result Demonstrates how to use the EarlyStopping package to explore implicit regularization.
Improved Fréchet regression tackles noise and multicollinearity.
problem Addressing noise and multicollinearity in multi-label regression.
method Implicit regularization framework for explicit modeling of relationships.
result Effective modeling of complex dependencies without introducing biases.
SNN architecture shows gradient descent converges to regularized solution in matrix sensing problems.
problem Understanding implicit regularization in neural networks for matrix sensing.
method Developed Spectral Neural Networks (SNN) for matrix learning problems, rigorously demonstrating implicit regularization.
result Gradient descent converges to the solution of a regularized learning problem in matrix sensing problems.
Gradient filters track moving parameters under noisy data and misspecification.
problem Tracking multidimensional time-varying parameters under noisy observations and model misspecification.
method Gradient-based filters update parameters using the gradient of a postulated objective function, evaluated at either the predicted or updated parameters.
result Novel sufficient conditions for exponential stability of the filtered parameter path, and finite-sample and asymptotic mean squared error bounds.
This work uses PAC-Bayes for structured prediction with ILE, yielding insights and algorithms.
problem Structured prediction with interdependent outputs and implicit loss embeddings.
method PAC-Bayes perspective applied to ILE framework, deriving generalization bounds and learning algorithms.
result Two learning algorithms derived from PAC-Bayes bounds, analyzed and implemented.
Recently there has been a surge of interest in understanding implicit regularization properties of iterative gradient-based optimization algorithms. In this paper, we study the statistical guarantees on the excess risk achieved by early-stopped unconstrained mirror descent algorithms applied to the unregularized empiri…
A tuning-free method recovers jointly sparse signals in MMV using implicit regularization.
problem Recovering jointly sparse signals in MMV with minimal tuning or prior knowledge.
method Reparameterizes MMV estimation matrix into decoupled factors and applies gradient descent to a least-squares objective.
result Gradient descent dynamics exhibit a momentum-like effect, converging towards an idealized row-sparse solution.
Gradient flow on ReLU networks converges to a simple model with few regions.
problem Understanding the dynamics of gradient flow in shallow ReLU networks.
method Analysis of gradient flow dynamics on univariate ReLU neural networks.
result Gradient flow converges to a network with at most O(r) linear regions.
New TD algorithms stabilize RL tasks by reformulating updates into fixed point equations.
problem TD learning's sensitivity to step size specification.
method Implicit TD algorithms reformulate TD updates into fixed point equations.
result Implicit TD algorithms are more stable and less sensitive to step size.
The paper studies implicit regularization in over-parameterized models for high-dimensional data.
problem Understanding implicit regularization in over-parameterized models for high-dimensional data.
method The paper designs regularization-free algorithms for the high-dimensional single index model and provides theoretical guarantees for the induced implicit regularization phenomenon.
result The proposed methods achieve minimax optimal statistical rates of convergence and outperform classical methods with explicit regularization.
An optimal control problem associated with the dynamics of the orientation of a bipolar molecule in the plane can be understood by means of tools in differential geometry. For first time in the literature k-symplectic formalism is used to provide the optimal control problems associated to some families of partial dif…
Our knowledge about the evolution of guarantee network in downturn period is limited due to the lack of comprehensive data of the whole credit system. Here we analyze the dynamic Chinese guarantee network constructed from a comprehensive bank loan dataset that accounts for nearly 80% total loans in China, during 01/200…
New algorithm for computing Wasserstein barycenters with guarantees.
problem Computing Wasserstein barycenters with varying regularization strengths.
method Damped Sinkhorn iterations followed by exact maximization/minimization steps.
result First non-asymptotic convergence guarantees for approximating Wasserstein barycenters.
Enhances deep learning by boosting generalization and convergence.
problem Improving generalization and convergence in deep learning models.
method Implicit Regularization Enhancement (IRE) framework that decouples flat and sharp directions.
result IRE consistently improves generalization performance across various deep learning tasks and models.
New scaling laws explain deep learning performance growth.
problem Understanding neural network performance growth.
method Analyzed entire training dynamics of various architectures.
result Identified two dynamical scaling laws.
We propose a \textbf{uni}fied \textbf{f}ramework for \textbf{i}mplicit \textbf{ge}nerative \textbf{m}odeling (UnifiGem) with theoretical guarantees by integrating approaches from optimal transport, numerical ODE, density-ratio (density-difference) estimation and deep neural networks. First, the problem of implicit gene…
Bayesian autoencoders discover physics from noisy data.
problem Challenges in identifying governing equations and coordinates from noisy, low-data real-world data.
method Bayesian SINDy autoencoders with hierarchical Bayesian sparsifying prior and adaptive empirical Bayesian method.
result Better physics discovery with lower data and fewer training epochs, along with valid uncertainty quantification.
In implicit models, one often interpolates between sampled points in latent space. As we show in this paper, care needs to be taken to match-up the distributional assumptions on code vectors with the geometry of the interpolating paths. Otherwise, typical assumptions about the quality and semantics of in-between points…
We present an approach to obtain convergence guarantees of optimization algorithms for deep networks based on elementary arguments and computations. The convergence analysis revolves around the analytical and computational structures of optimization oracles central to the implementation of deep networks in machine lear…
This work addresses the problem of regret minimization in non-stochastic multi-armed bandit problems, focusing on performance guarantees that hold with high probability. Such results are rather scarce in the literature since proving them requires a large deal of technical effort and significant modifications to the sta…