Adam is shown not being able to converge to the optimal solution in certain cases. Researchers recently propose several algorithms to avoid the issue of non-convergence of Adam, but their efficiency turns out to be unsatisfactory in practice. In this paper, we provide new insight into the non-convergence issue of Adam …
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,657 papers · 148 categories
Trend · papers per month
48 results for “non-convergence”
This paper studies a curious phenomenon in learning energy-based model (EBM) using MCMC. In each learning iteration, we generate synthesized examples by running a non-convergent, non-mixing, and non-persistent short-run MCMC toward the current model, always starting from the same initial distribution such as uniform no…
This paper studies the market phenomenon of non-convergence between futures and spot prices in the grains market. We postulate that the positive basis observed at maturity stems from the futures holder's timing options to exercise the shipping certificate delivery item and subsequently liquidate the physical grain. In …
Polyak-Ruppert CLT for SA-Adam with momentum and non-convergent adaptive preconditioning
problem Adaptive optimizers combining momentum and non-convergent preconditioning
method Proving positive drift stability and a non-autonomous Polyak-Ruppert CLT for SA-Adam
result The iterate-marginal covariance is exactly the plain stochastic gradient descent (SGD) sandwich
Study shows non-convergence of short-maturity expansion in SABR model.
problem Analyzing convergence of SABR model option prices.
method Analyzing analyticity properties of the payoff function and using the McKean kernel.
result Short-maturity expansion for SABR model is asymptotic and non-convergent.
The paper constructs non-convergent solutions to Vafa-Witten equations with specific harmonic 2-form limits.
problem Constructing solutions to Vafa-Witten equations with non-zero mass term.
method Constructs divergent sequences of solutions, renormalizes them, and defines harmonic 2-form data sets.
result Defines an 'interesting' harmonic 2-form data set with specific properties.
This paper extends stability analysis to non-convergent neural network training.
problem Generalization of neural networks whose training does not converge to fixed points.
method Introduces statistical algorithmic stability (SAS) to study non-convergent algorithms and their generalization.
result Stability of non-convergent training dynamics correlates with generalization performance.
ADOPT optimizes Adam to converge with any β2 without bounded noise.
problem Non-convergence of Adam optimization algorithm.
method ADOPT removes current gradient from second moment estimate and changes momentum update order.
result ADOPT achieves optimal convergence rate of O(1 / √T) with any β2.
Improved sampling for Bayesian neural networks reduces vanishing acceptance rates and increases predictive accuracy.
problem Sampling inefficiency in Bayesian neural networks, especially with deep architectures and large datasets.
method Approximate blocked Gibbs sampling to partition and sample subgroups of parameters.
result Increased predictive accuracy and quantification of predictive uncertainty in classification tasks.
In this paper, we show that Generative Adversarial Networks (GANs) suffer from catastrophic forgetting even when they are trained to approximate a single target distribution. We show that GAN training is a continual learning problem in which the sequence of changing model distributions is the sequence of tasks to the d…
SGD avoids critical points on weakly convex functions.
problem Non-convergence of SGD to critical points on specific manifolds.
method Stochastic subgradient descent, Verdier stratification, angle condition.
result SGD converges to local minimizers on weakly convex functions.
SGD methods fail to converge to global minimizers in deep neural networks with ReLU activation.
problem Failure of SGD methods to converge to global minimizers in deep neural networks.
method Stochastic Gradient Descent (SGD) and its variants like Adam, RMSProp, etc.
result SGD methods fail to converge to global minimizers with high probability in deep neural networks with ReLU activation.
Survey of GANs challenges and solutions for better model design and optimization.
problem Challenges in training GANs, including mode collapse, non-convergence, and instability.
method Comprehensive survey of GANs design and optimization solutions, proposing a new taxonomy.
result Presentation of promising research directions in GANs.
In this paper, we shall study the Dirichlet problem for the minimal surfaces equation. We prove some results about the boundary behaviour of a solution of this problem. We describe the behaviour of a non-converging sequence of solutions in term of lines of divergence in the domain. Using this second result, we build so…
Study convergence of Yamabe flow on singular spaces with positive constant.
problem Analyzing convergence of Yamabe flow on singular spaces.
method Normalized Yamabe flow with positive Yamabe constant on pseudo-manifolds, including stratified spaces.
result Established convergence under low energy condition and investigated alternatives.
Optimal Dynamic Basis Tradingq-fin.PM
We study the problem of dynamically trading a futures contract and its underlying asset under a stochastic basis model. The basis evolution is modeled by a stopped scaled Brownian bridge to account for non-convergence of the basis at maturity. The optimal trading strategies are determined from a utility maximization pr…
CR Yamabe flow fails to converge on small deformations of the standard CR three-sphere.
problem CR Yamabe flow convergence
method Constructing a contact form with negative pseudohermitian mass
result CR Yamabe flow fails to converge on small deformations of the standard CR three-sphere
Markov chain Monte Carlo (MCMC) methods have not been broadly adopted in Bayesian neural networks (BNNs). This paper initially reviews the main challenges in sampling from the parameter posterior of a neural network via MCMC. Such challenges culminate to lack of convergence to the parameter posterior. Nevertheless, thi…
Generative Adversarial Network (GAN) is a current focal point of research. The body of knowledge is fragmented, leading to a trial-error method while selecting an appropriate GAN for a given scenario. We provide a comprehensive summary of the evolution of GANs starting from its inception addressing issues like mode col…
Stochastic Stein Discrepancies improve inference efficiency.
problem Intractable computation of Stein discrepancies.
method Subsampled approximations of Stein operators.
result Stochastic Stein Discrepancies inherit convergence properties of standard SDs.
Persistently trained EBMs generate images and estimate complex densities.
problem Challenges in ML learning for energy-based models, especially non-convergence of MCMC.
method Introduce diffusion data, learn a joint EBM through persistent training with enhanced sampling.
result First simultaneous achievement of stability, post-training image generation, and superior out-of-distribution detection for image data.
Developed a new algorithm to improve dynamic treatment regimens.
problem Non-convergence of Q-learning-based Q-shared algorithm in dynamic treatment regimens.
method Penalized Q-shared algorithm to address convergence issues.
result The penalized Q-shared algorithm converges and outperforms the original in various settings.
First-order optimization algorithms have been proven prominent in deep learning. In particular, algorithms such as RMSProp and Adam are extremely popular. However, recent works have pointed out the lack of ``long-term memory" in Adam-like algorithms, which could hamper their performance and lead to divergence. In our s…
Despite the growing prominence of generative adversarial networks (GANs), optimization in GANs is still a poorly understood topic. In this paper, we analyze the "gradient descent" form of GAN optimization i.e., the natural setting where we simultaneously take small gradient steps in both generator and discriminator par…
Belief Propagation (BP) is one of the most popular methods for inference in probabilistic graphical models. BP is guaranteed to return the correct answer for tree structures, but can be incorrect or non-convergent for loopy graphical models. Recently, several new approximate inference algorithms based on cavity distrib…
SGD fails to converge for deep ReLU networks with limited random initializations.
problem SGD convergence in deep neural networks with limited random initializations.
method Analysis of four discretization parameters: network architecture, training data, gradient steps, and random initializations.
result SGD fails to converge for ReLU networks with depth much larger than width.
Generative Adversarial Networks (GANs) have shown remarkable results in modeling complex distributions, but their evaluation remains an unsettled issue. Evaluations are essential for: (i) relative assessment of different models and (ii) monitoring the progress of a single model throughout training. The latter cannot be…
Generative adversarial networks (GANs) are notoriously difficult to train and the reasons underlying their (non-)convergence behaviors are still not completely understood. By first considering a simple yet representative GAN example, we mathematically analyze its local convergence behavior in a non-asymptotic way. Furt…
Approximate Markov chain Monte Carlo (MCMC) offers the promise of more rapid sampling at the cost of more biased inference. Since standard MCMC diagnostics fail to detect these biases, researchers have developed computable Stein discrepancy measures that provably determine the convergence of a sample to its target dist…
In recent years, Generative Adversarial Networks (GANs) have received significant attention from the research community. With a straightforward implementation and outstanding results, GANs have been used for numerous applications. Despite the success, GANs lack a proper theoretical explanation. These models suffer from…
This paper tackles sampling issues in latent space EBMs by introducing diffusion-based amortization.
problem Degenerate MCMC sampling quality hinders latent space EBM learning and generation quality.
method Introduces diffusion-based amortization for long-run MCMC sampling.
result The learned amortization of MCMC is a valid long-run MCMC sampler.
This study compares parallel SMC and MCMC for Bayesian deep learning, showing SMC parallel is faster.
problem Efficiently performing Bayesian deep learning with parallel computing.
method Compared sequential Monte Carlo (SMC) and Markov chain Monte Carlo (MCMC) in parallel settings.
result Parallel SMC achieves similar convergence as a single SMC but with reduced communication time.
Paper studies stochastic optimization methods with momentum, proving convergence and avoiding traps.
problem Optimizing non-convex functions with momentum.
method Unified analysis of stochastic gradient descent variants, including S-NAG and Adam.
result Convergence to critical points and avoidance of undesired critical points like local maxima or saddle points.
Stochastic gradient descent approximates Gaussian process posteriors efficiently.
problem Efficiently sampling from Gaussian process posteriors with limited computational resources.
method Developed stochastic gradient optimization objectives for sampling from Gaussian process posteriors.
result Stochastic gradient descent produces accurate predictive distributions, even in non-convergent cases.
Training deep neural networks requires intricate initialization and careful selection of learning rates. The emergence of stochastic gradient optimization methods that use adaptive learning rates based on squared past gradients, e.g., AdaGrad, AdaDelta, and Adam, eases the job slightly. However, such methods have also …
In complex simulation environments, certain parameter space regions may result in non-convergent or unphysical outcomes. All parameters can therefore be labeled with a binary class describing whether or not they lead to valid results. In general, it can be very difficult to determine feasible parameter regions, especia…
Recently, there is a growing interest in the study of median-based algorithms for distributed non-convex optimization. Two prominent such algorithms include signSGD with majority vote, an effective approach for communication reduction via 1-bit compression on the local gradients, and medianSGD, an algorithm recently pr…
Symmetric (Optimistic) Natural Policy Gradient for Multi-agent Learning with Parameter Convergencemath.OC
New NPG variants ensure parameter convergence in multi-agent learning.
problem Non-convergence of parameters in NPG for multi-agent learning.
method Proposed variants of NPG for multi-agent learning scenarios.
result Global last-iterate parameter convergence guarantees in various multi-agent learning settings.
Learning an efficient update rule from data that promotes rapid learning of new tasks from the same distribution remains an open problem in meta-learning. Typically, previous works have approached this issue either by attempting to train a neural network that directly produces updates or by attempting to learn better i…
Adaptive inference for -estimators in bandit data with model misspecification.
problem Challenges in off-policy inference for adaptively collected bandit data with a misspecified model.
method A novel approach to define a projected solution over a stationary evaluation policy, stabilizing variance with flexible methods.
result Valid inference for -estimators in adaptive settings, even with unstable treatment policies.
Koopman mode analysis applied to neural networks for training optimization.
problem Optimizing neural network training, identifying issues, and speeding up learning.
method Koopman operator analysis of neural network dynamics.
result Spectral analysis of Koopman operator aids in determining network depth, initialization quality, and training termination.
Network Embedding is the task of learning continuous node representations for networks, which has been shown effective in a variety of tasks such as link prediction and node classification. Most of existing works aim to preserve different network structures and properties in low-dimensional embedding vectors, while neg…
A new method learns latent space normalizing flow for approximate inference in generator models.
problem Approximate inference in generator models with complex posterior distributions.
method Jointly learns latent space normalizing flow and generator model using MCMC-based maximum likelihood.
result The short-run Langevin flow approximates the posterior and aligns with the normalizing flow prior.
Recent years have witnessed significant advances in reinforcement learning (RL), which has registered great success in solving various sequential decision-making problems in machine learning. Most of the successful RL applications, e.g., the games of Go and Poker, robotics, and autonomous driving, involve the participa…
The paper tackles temporal coverage bias in financial panel data, proposing a structuring framework to correct for incomplete histories.
problem Incomplete histories of financial instruments lead to biased panel data.
method Formalizes the problem and proposes a coverage-aware structuring framework using structured metadata and an availability matrix.
result The framework reveals substantial distortions in return dynamics and volatility when naive temporal alignment is used.
Bayesian Optimization tackles hidden constraints in architecture optimization.
problem Optimizing system architectures with hidden constraints using expensive physics-based simulations.
method Surrogate-based optimization with Gaussian Process models, including strategies for handling failed evaluations.
result Best performance achieved with a mixed-discrete GP predicting Probability of Viability (PoV) and minimum PoV threshold selection.
Linearized attention fails to converge to NTK limit even at large widths.
problem Understanding the convergence of attention mechanisms to the kernel regime.
method Analyzes linearized attention and its relationship to the NTK limit, considering practical widths and conditions.
result Linearized attention does not converge to its NTK limit at any practical width, revealing a fundamental trade-off.
This paper clarifies Bitcoin's volatility and predictability across daily, weekly, and monthly scales.
problem Clarify Bitcoin's volatility and predictability across different time scales.
method Using daily, weekly, and monthly closing prices and log-returns data, analyze volatility and predictability.
result Bitcoin exhibits high volatility and high predictability, with different behaviors at different time scales.