arXiv research

Truncated backpropagation through time (TBPTT) is a popular method for learning in recurrent neural networks (RNNs) that saves computation and memory at the cost of bias by truncating backpropagation after a fixed number of lags. In practice, choosing the optimal truncation length is difficult: TBPTT will not converge …

Truncated SGD with heavy-tailed noise eliminates sharp local minima.

New DP framework using data truncation for efficient estimation.

Efficiently estimate Boolean product distribution parameters from truncated samples.

We study functions whose truncations are convex or quasiconvex.

Stochastic gradient descent (SGD) is commonly used for optimization in large-scale machine learning problems. Langford et al. (2009) introduce a sparse online learning method to induce sparsity via truncated gradient. With high-dimensional sparse data, however, the method suffers from slow convergence and high variance…

Derives equations for deep learning biases and weights, showing data complexity reduction.

Recent work has demonstrated the effectiveness of gradient descent for directly recovering the factors of low-rank matrices from random linear measurements in a globally convergent manner when initialized properly. However, the performance of existing algorithms is highly sensitive in the presence of outliers that may …

Paper proposes approximate Stein classes for efficient truncated density estimation.

Optimal algorithm learns Gaussian under halfspace truncation with minimal samples.

A new method reduces variance in PG methods for RL, improving efficiency and convergence.

The paper analyzes and mitigates biases in scalable Gaussian Process methods.

Bilevel optimization has been recently revisited for designing and analyzing algorithms in hyperparameter tuning and meta learning tasks. However, due to its nested structure, evaluating exact gradients for high-dimensional problems is computationally challenging. One heuristic to circumvent this difficulty is to use t…

UDN adapts depth to data complexity, outperforming standard neural networks.

Learning with a {\it convex loss} function has been a dominating paradigm for many years. It remains an interesting question how non-convex loss functions help improve the generalization of learning with broad applicability. In this paper, we study a family of objective functions formed by truncating traditional loss f…

PES method reduces bias in gradient estimation for unrolled graphs.

A large class of machine learning techniques requires the solution of optimization problems involving spectral functions of parametric matrices, e.g. log-determinant and nuclear norm. Unfortunately, computing the gradient of a spectral function is generally of cubic complexity, as such gradient descent methods are rath…

AOPU stabilizes NN training by approximating natural gradient, improving stability and convergence.

As in standard linear regression, in truncated linear regression, we are given access to observations $(A_i, y_i)_i$ whose dependent variable equals $y_i= A_i^{\rm T} \cdot x^* + η_i$, where $x^*$ is some fixed unknown vector of interest and $η_i$ is independent noise; except we are only given an observation if its dep…

Deep learning has shown that learned functions can dramatically outperform hand-designed functions on perceptual tasks. Analogously, this suggests that learned optimizers may similarly outperform current hand-designed optimizers, especially for specific problems. However, learned optimizers are notoriously difficult to…

We propose a new framework for Hamiltonian Monte Carlo (HMC) on truncated probability distributions with smooth underlying density functions. Traditional HMC requires computing the gradient of potential function associated with the target distribution, and therefore does not perform its full power on truncated distribu…

This paper presents a new algorithm, termed \emph{truncated amplitude flow} (TAF), to recover an unknown vector $\bm{x}$ from a system of quadratic equations of the form $y_i=|\langle\bm{a}_i,\bm{x}\rangle|^2$, where $\bm{a}_i$'s are given random measurement vectors. This problem is known to be \emph{NP-hard} in genera…

We deal with the efficient parallelization of Bayesian global optimization algorithms, and more specifically of those based on the expected improvement criterion and its variants. A closed form formula relying on multivariate Gaussian cumulative distribution functions is established for a generalized version of the mul…

ES-Single uses ES to estimate gradients in unrolled graphs, reducing variance and improving performance.

A new method optimizes diffusion models with recursive likelihood ratios.

State construction is important for learning in partially observable environments. A general purpose strategy for state construction is to learn the state update using a Recurrent Neural Network (RNN), which updates the internal state using the current internal state and the most recent observation. This internal state…

The problem of an arbitrary truncated Levy flight description using the method of cumulant approach has been solved. The set of cumulants of the truncated Levy distribution given the assumption of arbitrary truncation has been found. The influence of truncation shape on the truncated Levy flight properties in the Gauss…

A new method distills datasets more efficiently and effectively.

A novel approach termed \emph{stochastic truncated amplitude flow} (STAF) is developed to reconstruct an unknown $n$-dimensional real-/complex-valued signal $\bm{x}$ from $m$ `phaseless' quadratic equations of the form $ψ_i=|\langle\bm{a}_i,\bm{x}\rangle|$. This problem, also known as phase retrieval from magnitude-onl…

We propose a communicationally and computationally efficient algorithm for high-dimensional distributed sparse learning. At each iteration, local machines compute the gradient on local data and the master machine solves one shifted $l_1$ regularized minimization problem. The communication cost is reduced from constant …

In the paper "On Truncated Variation of Brownian Motion with Drift" (Bull. Pol. Acad. Sci. Math. 56 (2008), no.4, 267 - 281) we defined truncated variation of Brownian motion with drift, $W_t = B_t + μt, t\geq 0,$ where $(B_t)$ is a standard Brownian motion. Truncated variation differs from regular variation by neglect…

Paper tackles heavy-tailed data without finite variance, proposing robust risk minimization.

New method for constructing truncated vine copulas.

Non-negative matrix factorization (NMF) minimizes the Euclidean distance between the data matrix and its low rank approximation, and it fails when applied to corrupted data because the loss function is sensitive to outliers. In this paper, we propose a Truncated CauchyNMF loss that handle outliers by truncating large e…

RTRL optimizes long sequences without truncation, converging to loss minima.

Solving logistic regression with L1-regularization in distributed settings is an important problem. This problem arises when training dataset is very large and cannot fit the memory of a single machine. We present d-GLMNET, a new algorithm solving logistic regression with L1-regularization in the distributed settings. …

Paper defines new risk measures for elliptical distributions.

A quantum reinforcement learning algorithm reduces sample complexity.

SUMO provides unbiased log marginal likelihood estimation for latent variable models.

In structured output prediction tasks, labeling ground-truth training output is often expensive. However, for many tasks, even when the true output is unknown, we can evaluate predictions using a scalar reward function, which may be easily assembled from human knowledge or non-differentiable pipelines. But searching th…

One of the central goals of Recurrent Neural Networks (RNNs) is to learn long-term dependencies in sequential data. Nevertheless, the most popular training method, Truncated Backpropagation through Time (TBPTT), categorically forbids learning dependencies beyond the truncation horizon. In contrast, the online training …

Unified framework for mean testing under truncation bias.

Score matching method improves density estimation for truncated data on manifolds.

This paper introduces the Metric-Free Natural Gradient (MFNG) algorithm for training Boltzmann Machines. Similar in spirit to the Hessian-Free method of Martens [8], our algorithm belongs to the family of truncated Newton methods and exploits an efficient matrix-vector product to avoid explicitely storing the natural g…

New methods recover best rank-r approximations from few entries.

Truncated densities are probability density functions defined on truncated domains. They share the same parametric form with their non-truncated counterparts up to a normalizing constant. Since the computation of their normalizing constants is usually infeasible, Maximum Likelihood Estimation cannot be easily applied t…

The method approximates stationary distributions of Markov models by truncating irrelevant states.

We developed efficient methods to compute gradients for Neural SDEs, improving training speed and accuracy.