A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
New bounds for SGLD show error decreases with more data.
New algorithm tackles nonconvex machine learning problems with adaptive normalization and independent sampling.
Tensor programs prove neural network limits for any architecture.
New bounds for neural networks without loss boundedness assumption.
DDICA separates nonlinear mixed signals robustly.
The paper relaxes assumptions for analyzing stochastic optimization algorithms.
OT-ICA uses optimal transport to find independent components, outperforming traditional methods.
AdaGrad-Norm achieves optimal convergence rates for non-convex objectives without tuning.
The options framework is a popular approach for building temporally extended actions in reinforcement learning. In particular, the option-critic architecture provides general purpose policy gradient theorems for learning actions from scratch that are extended in time. However, past work makes the key assumption that ea…
We construct exhaustion and cut-off functions with controlled gradient and Laplacian on manifolds with Ricci curvature bounded from below by a (possibly unbounded) nonpositive function of the distance from a fixed reference point, without any assumptions on the topology or the injectivity radius. Along the way we prove…
This paper proposes a novel proximal-gradient algorithm for a decentralized optimization problem with a composite objective containing smooth and non-smooth terms. Specifically, the smooth and nonsmooth terms are dealt with by gradient and proximal updates, respectively. The proposed algorithm is closely related to a p…
A new algorithm estimates sparse gradients on graphs with improved risk bounds.
We investigate 1) the rate at which refined properties of the empirical risk---in particular, gradients---converge to their population counterparts in standard non-convex learning tasks, and 2) the consequences of this convergence for optimization. Our analysis follows the tradition of norm-based capacity control. We p…
A new gradient estimator for online optimization with two function evaluations.
We analyze the variance of stochastic gradients along negative curvature directions in certain non-convex machine learning models and show that stochastic gradients exhibit a strong component along these directions. Furthermore, we show that - contrary to the case of isotropic noise - this variance is proportional to t…
We obtain a compactness result for Fano manifolds and Kähler Ricci flows. Comparing to the more general Riemannian versions by Anderson and Hamilton, in this Fano case, the curvature assumption is much weaker and is preserved by the Kähler Ricci flows. One assumption is the boundedness of the Ricci potential and the ot…
Neurosymbolic predictors fail to model uncertainty under independence assumption.
A new hybrid algorithm reduces stochastic gradient evaluations for nonconvex optimization.
We study the problem of sampling from a probability distribution on $\rset^d$ which has a density \wrt\ the Lebesgue measure known up to a normalization factor $x \mapsto \rme^{-U(x)} / \int_{\rset^d} \rme^{-U(y)} \rmd y$. We analyze a sampling method based on the Euler discretization of the Langevin stochastic dif…
Several recent trends in machine learning theory and practice, from the design of state-of-the-art Gaussian Process to the convergence analysis of deep neural nets (DNNs) under stochastic gradient descent (SGD), have found it fruitful to study wide random neural networks. Central to these approaches are certain scaling…
The standard linear and logistic regression models assume that the response variables are independent, but share the same linear relationship to their corresponding vectors of covariates. The assumption that the response variables are independent is, however, too strong. In many applications, these responses are collec…
This is the first of a series of papers devoted to a thorough analysis of the class of gradient flows in a metric space that can be characterized by Evolution Variational Inequalities. We present new results concerning the structural properties of solutions to the formulation, such as co…
Gradient descent learns a single neuron without knowing the relationship between inputs and labels.
We describe a layer-by-layer algorithm for training deep convolutional networks, where each step involves gradient updates for a two layer network followed by a simple clustering algorithm. Our algorithm stems from a deep generative model that generates mages level by level, where lower resolution images correspond to …
Recently, there is a growing interest in the study of median-based algorithms for distributed non-convex optimization. Two prominent such algorithms include signSGD with majority vote, an effective approach for communication reduction via 1-bit compression on the local gradients, and medianSGD, an algorithm recently pr…
In recent years, the mean field theory has been applied to the study of neural networks and has achieved a great deal of success. The theory has been applied to various neural network structures, including CNNs, RNNs, Residual networks, and Batch normalization. Inevitably, recent work has also covered the use of dropou…
Study shows convergence of stochastic gradient method for unregularized Wasserstein optimization.
Gradient descent struggles with learning a single neuron with bias.
To reduce the long training time of large deep neural network (DNN) models, distributed synchronous stochastic gradient descent (S-SGD) is commonly used on a cluster of workers. However, the speedup brought by multiple workers is limited by the communication overhead. Two approaches, namely pipelining and gradient spar…
Paper relaxes independence assumption for non-centered data.
New research challenges the independence assumption in neurosymbolic learning, leading to overconfident predictions and unrepresentable uncertainty.
It is well known that Markov chain Monte Carlo (MCMC) methods scale poorly with dataset size. A popular class of methods for solving this issue is stochastic gradient MCMC. These methods use a noisy estimate of the gradient of the log posterior, which reduces the per iteration computational cost of the algorithm. Despi…
New SGD analysis for nonconvex optimization finds optimal rates.
Gradient flows for surface energies with tensor fields are derived and analyzed.
The rise of deep learning in recent years has brought with it increasingly clever optimization methods to deal with complex, non-linear loss functions. These methods are often designed with convex optimization in mind, but have been shown to work well in practice even for the highly non-convex optimization associated w…
We present a novel algorithm for overcomplete independent components analysis (ICA), where the number of latent sources k exceeds the dimension p of observed variables. Previous algorithms either suffer from high computational complexity or make strong assumptions about the form of the mixing matrix. Our algorithm does…
Efficient spatio-temporal Gaussian process inference method.
Proposes a weaker faithfulness assumption for causal discovery.
Empirical studies show that gradient-based methods can learn deep neural networks (DNNs) with very good generalization performance in the over-parameterization regime, where DNNs can easily fit a random labeling of the training data. Very recently, a line of work explains in theory that with over-parameterization and p…
Gradient flow on ReLU networks converges to a simple model with few regions.
Although neural networks are routinely and successfully trained in practice using simple gradient-based methods, most existing theoretical results are negative, showing that learning such networks is difficult, in a worst-case sense over all data distributions. In this paper, we take a more nuanced view, and consider w…
Most existing works on disentangled representation learning are solely built upon an marginal independence assumption: all factors in disentangled representations should be statistically independent. This assumption is necessary but definitely not sufficient for the disentangled representations without additional induc…
A fast MCMC sampler for sparse Bayesian inference.
Paper addresses fault-tolerance in distributed machine learning with stochastic gradient descent.
The paper reveals low-rank structure in neural network gradients, influenced by data and model parameters.
Independent component analysis (ICA) is a powerful method for blind source separation based on the assumption that sources are statistically independent. Though ICA has proven useful and has been employed in many applications, complete statistical independence can be too restrictive an assumption in practice. Additiona…
New algorithm resists Byzantine attacks in distributed SGD for heterogeneous data.
The multiresolution Gaussian process (GP) has gained increasing attention as a viable approach towards improving the quality of approximations in GPs that scale well to large-scale data. Most of the current constructions assume full independence across resolutions. This assumption simplifies the inference, but it under…