New control theory approach stabilizes GANs training.
problem Stability issues in GANs training.
method Control theory applied to GANs function space dynamics.
result Effective stabilization of GANs training with CLC and squared regularizer.
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
169,341 papers · 148 categories
Trend · papers per month
48 results for “Stability in training”
Weight decay stabilizes training dynamics by slowing progressive sharpening.
problem Understanding how weight decay affects training stability in deep learning models.
method Analyzing weight decay effects at the Edge of Stability, developing a mathematical framework.
result Weight decay dampens oscillations and stabilizes sharpness in CNNs, causing a phase transition in MLPs.
Stability training improves deep neural networks' robustness without data augmentation.
problem Improving deep neural networks' robustness against input perturbations.
method Stability training as an alternative to data augmentation.
result Stability training outperforms data augmentation in specific transformations and offers improved robustness against a broader range of distortions.
GNMR controls runtime stability in low-precision language model training.
problem Efficient low-precision training faces numerical risks at specific operators.
method GNMR compares gradient norms to historical means, applying bounded recovery actions.
result GNMR preserves high-fidelity quality with sparse, budgeted recovery.
Training avoids edge of stability by aligning Jacobian matrices.
problem Training neural networks on the edge of stability causes inaccuracies.
method Used an exponential Euler solver to prevent entering the edge of stability.
result Alignment of Jacobian matrices causes sharpness increase in Hessian.
Gradient descent at edge of stability stabilizes implicitly, following projected gradient descent.
problem Gradient descent's stability and sharpness behavior at the edge of instability.
method Cubic Taylor expansion analysis of gradient descent dynamics.
result Gradient descent at edge of stability implicitly follows projected gradient descent.
AOPU stabilizes NN training by approximating natural gradient, improving stability and convergence.
problem Stability and interpretability in online NN training for industrial soft sensors.
method AOPU truncates gradient backpropagation, optimizing trackable parameters, and approximating natural gradient.
result AOPU achieves stable convergence and superior performance on chemical process datasets.
Improved stability and generalization for blackbox learned optimizers.
problem Stability and generalization issues in blackbox learned optimizers.
method Investigation using dynamical systems, modifications to optimizer architecture and meta-training procedure.
result Improved stability and generalization of learned optimizers.
This paper extends stability analysis to non-convergent neural network training.
problem Generalization of neural networks whose training does not converge to fixed points.
method Introduces statistical algorithmic stability (SAS) to study non-convergent algorithms and their generalization.
result Stability of non-convergent training dynamics correlates with generalization performance.
Spectral normalization stabilizes GAN training for better image generation.
problem Stability issues in GAN training.
method Spectral normalization technique to stabilize discriminator training.
result Spectrally normalized GANs (SN-GANs) generate better quality images.
Gradient descent on neural nets often operates at the Edge of Stability, where loss behavior is complex but loss decreases over time.
problem Understanding the optimization dynamics of neural networks at the Edge of Stability.
method Empirical demonstration of gradient descent behavior in neural network training.
result Gradient descent on neural networks typically occurs at the Edge of Stability, where loss behavior is non-monotonic but loss decreases over time.
New insights into why Transformers are hard to train, leading to a new method to stabilize them.
problem Understanding and stabilizing the training of Transformers.
method Analyzing the amplification effect in Transformer models and proposing Admin to stabilize training.
result Admin stabilizes early stage training and improves performance.
Framework increases neural net stability through adversarial training and robust optimization.
problem Increasing local stability of neural nets to improve robustness against adversarial attacks.
method Alternating minimization-maximization procedure with perturbed examples.
result Improves robustness and accuracy of neural nets against adversarial examples.
We explore in some detail the notion of algorithmic stability as a viable framework for analyzing the generalization error of learning algorithms. We introduce the new notion of training stability of a learning algorithm and show that, in a general setting, it is sufficient for good bounds on generalization error. In t…
On Lyapunov Exponents for RNNs: Understanding Information Propagation Using Dynamical Systems Toolscs.LG
Lyapunov exponents help understand RNN stability.
problem Optimizing RNNs is sensitive to various parameters.
method Use Lyapunov exponents as dynamical system tools.
result Lyapunov spectrum measures training stability.
Bidirectional bounds stabilize training of energy-based models.
problem Training energy-based models is difficult and prone to instability.
method Propose bidirectional bounds linking to gradient penalty and Jacobi-determinant estimator.
result Significant stabilization and high-quality density estimation achieved.
Stabilizes complex systems using diffusion models trained on Lyapunov functions.
problem Generating stabilizing controllers for complex dynamical systems.
method Trains a diffusion model on pairs of asymptotically stable vector fields and their Lyapunov functions to identify the closest stable field and adjust control functions.
result Efficient and rapid stabilization of unseen systems, showcasing generalizability.
We improve neural network robustness verification by training for faster stability.
problem Efficient verification of adversarial robustness in deep networks.
method Co-design of weight sparsity and ReLU stability to simplify verification.
result Improving ReLU stability leads to a 4-13x speedup in verification times.
Modular NNs improve training speed and stability.
problem Complexity in NNs with many parameters or intricate architectures.
method Decompose NN into control and functional modules.
result Modular NNs outperform monolithic ones in training speed and stability.
Graph neural network predicts grasp stability from tactile sensor data.
problem Predicting grasp stability from tactile sensor data.
method Graph Convolutional Network (GCN) trained on tactile sensor data.
result Graph neural network effectively predicts grasp stability.
The study identifies conditions under which algorithmic stability explains generalization in interpolating learning systems.
problem Understanding when algorithmic stability explains generalization in interpolating learning systems.
method Modeling training as a function-space trajectory and measuring sensitivity to single-sample perturbations.
result There exist interpolating regimes with small risk where contractive sensitivity cannot hold, showing that stability is not a universal explanation.
Unbalanced GANs stabilize GAN training by pre-training the generator with VAE.
problem Stable training of GANs to avoid mode collapses and improve image quality.
method Pre-train GAN generator with VAE, balance generator and discriminator training, prevent discriminator's early convergence.
result Unbalanced GANs reduce mode collapses and outperform ordinary GANs in stability, convergence, and image quality.
The paper derives uniform stability-based coverage bounds for conformal prediction methods.
problem Establishing theoretical guarantees for conformal prediction methods.
method Uniform stability perspective applied to full-conformal, jackknife+, and CV+ prediction regions.
result Coverage bounds for finite-dimensional models derived using a concentration argument.
Stochastic ensembling improves stability in training Generative Adversarial Networks.
problem Oscillation, mode collapse, and imbalance between generator and discriminator.
method Stochastic ensembling of neural networks.
result Improves stability and learning capacity of GANs.
Unsupervised learning finds features for better generalization in block stability prediction.
problem Improving generalization to unseen scenarios in block stability prediction.
method Training an unsupervised model to predict future frames of stable and unstable block configurations.
result Unsupervised model features support extrapolating stability prediction to unseen block configurations.
Study stabilizes adversarial training in neural networks over infinite-dimensional spaces.
problem Stability issues in adversarial training of neural networks.
method Functional analysis of minimax optimization over infinite-dimensional spaces of continuous functions and probability measures.
result Convergence property of minimax problems under certain conditions, interpreted as stabilization techniques.
The study examines instance label stability in MIL classifiers trained on global image annotations.
problem Instance labels in MIL classifiers can be unstable, leading to incorrect fine-grained annotations.
method Investigated instance stability on 5 datasets, proposing an unsupervised measure.
result A performance-stability trade-off can be made when comparing MIL classifiers.
MAML++ improves MAML's stability and performance.
problem Stability and high generalization in few-shot learning.
method Various modifications to MAML to stabilize and improve performance.
result Significant improvements in stability, generalization, and computational efficiency.
Free adversarial training reduces the generalization gap compared to vanilla method.
problem Improving generalization in adversarial training.
method Analysis of algorithmic stability in free adversarial training.
result Free adversarial training shows a lower generalization gap.
Improved stable RNNs trained faster with less expressibility trade-off.
problem Stable recurrent neural networks are hard to train without sacrificing expressibility.
method Implicit model structure with contraction analysis for stable models.
result Significant increase in training speed and model performance.
New rules found to maintain neural network performance in asynchronous training.
problem Asynchronous training leads to degradation in generalization.
method Examined dynamical stability, derived rules for learning rate adjustment.
result Learning rate should be inversely proportional to delay for high delay values.
Stabilizes GAN training with limited data.
problem Overfitting in GANs with scarce data.
method Adaptive discriminator augmentation.
result Good results possible with few thousand images.
Kernel-guided training stabilizes GANs by controlling discrepancies.
problem Stability and interpretability issues in GANs.
method Kernel-based regularization to control discrepancies in GAN loss function.
result Theoretical guarantees on stability of the training dynamics.
Kernel-guided training stabilizes GANs by controlling discrepancies.
problem Stability and interpretability issues in GANs.
method Kernel-based regularization to control discrepancies in GAN loss function.
result Theoretical guarantees on stability of the training dynamics.
New method improves GAN training stability and quality.
problem Improving training stability and sample quality in GANs.
method Proposes a new method for Lipschitz continuity in GANs that is efficient and unbiased.
result Demonstrates the effectiveness of the new method in various GAN training scenarios.
Improved GAN training with WN over BN, achieving better quality and stability.
problem Training instability and quality issues in GANs.
method Introduced Weight Normalization (WN) and a new objective measure for evaluation.
result Training using WN is superior to BN, achieving lower reconstruction loss and better qualitative results.
ALRC clips learning rates to stabilize neural network training.
problem Stability issues in training with small batch sizes and high learning rates.
method Adaptive learning rate clipping (ALRC) to limit losses.
result ALRC decreases errors in unstable training conditions.
The study analyzes sharpness dynamics in neural networks, revealing mechanisms and conditions.
problem Understanding sharpness in neural network training.
method Fixed point analysis and edge of stability analysis in a simplified 2-layer linear network.
result Reveals mechanisms behind sharpness trends, conditions for edge of stability, and a period-doubling route to chaos.
Stability of the Decoupled Extended Kalman Filter Learning Algorithm in LSTM-Based Online Learningcs.LG
DEKF maintains stability in LSTM learning with bounded perturbations.
problem Stability of DEKF in LSTM-based online learning.
method Modeling DEKF as a perturbed extended Kalman filter and deriving stability conditions.
result DEKF learns LSTM parameters with similar stability properties to the global extended Kalman filter.
Spectral normalization stabilizes GANs by controlling gradient explosion and vanishing.
problem Stability and sample quality issues in GAN training.
method Spectral normalization controls gradient explosion and vanishing, improving GAN training stability and sample quality.
result Bidirectional Scaled Spectral Normalization (BSSN) outperforms standard spectral normalization in sample quality and training stability.
The paper studies stability of generative models trained on mixed data.
problem Training generative models on mixed datasets (real and synthetic data).
method Developed a framework to rigorously study stability under specific conditions.
result Proved the stability of iterative training under certain conditions.
Stabilizes GANs by unrolling discriminator optimization.
problem Mode collapse and unstable training in GANs.
method Defines generator objective with unrolled discriminator optimization.
result Solves mode collapse, stabilizes training, increases diversity.
New stability conditions for ZO methods reveal unique regularization effects.
problem Understanding optimization dynamics of ZO methods in deep learning.
method Explicit step size conditions and stability bounds derived for ZO methods.
result ZO methods operate near the edge of stability, with regularization effects specific to Hessian trace vs. eigenvalue.
Sharp scaling factor τ stabilizes deep ResNets and improves convergence.
problem Ensuring stable training of deep ResNets with gradient descent.
method Scaling the parametric branch by to guarantee stable training.
result Gradient descent finds global minima for properly over-parameterized ResNets.
We introduce a new parameterization method for deep learning layers using spectral tensor train decomposition.
problem Efficiency and stability in deep learning models with weight matrix compression.
method Spectral Tensor Train Parameterization (STTP) of weight matrices.
result Improved compression and training stability in neural networks.
Improved GAN training stability through clipping and reweighting.
problem Inconsistent GAN training leading to inferior performance.
method Proposes a variational GAN framework with probability ratio clipping and sample reweighting.
result Significantly improved performance across various GAN tasks.
Intervention loss stabilizes GAN training and reduces mode collapse.
problem Stabilizing GAN training and mitigating mode collapse.
method Intervention loss regularization term introduced into GAN objective.
result Improved GAN training stability and reduced mode collapse.
A new method speeds up and stabilizes deep CNN training.
problem Training deep convolutional neural networks (CNNs) is slow and unstable.
method Nonlinearity generation (NG) method modifies activation functions to stabilize training.
result The method speeds up and stabilizes training of deep CNNs.