arXiv research

This paper proves subsampled Newton methods work for high-dimensional data.

Efficient methods for training deep neural networks using subsampled Gauss-Newton and natural gradient.

The paper studies the solution of stochastic optimization problems in which approximations to the gradient and Hessian are obtained through subsampling. We first consider Newton-like methods that employ these approximations and discuss how to coordinate the accuracy in the gradient and Hessian to yield a superlinear ra…

Sketching, a dimensionality reduction technique, has received much attention in the statistics community. In this paper, we study sketching in the context of Newton's method for solving finite-sum optimization problems in which the number of variables and data points are both large. We study two forms of sketching that…

SVRN accelerates Newton methods by reducing variance and improving performance.

We propose a fast proximal Newton-type algorithm for minimizing regularized finite sums that returns an $ε$-suboptimal point in $\tilde{\mathcal{O}}(d(n + \sqrt{κd})\log(\frac{1}ε))$ FLOPS, where $n$ is number of samples, $d$ is feature dimension, and $κ$ is the condition number. As long as $n > d$, the proposed method…

Bayesian coresets improved with random sampling and quasi-Newton optimization.

Proposes STRON method for large-scale machine learning problems.

Stochastic second-order methods converge fast under interpolation conditions.

A new Bayesian filtering method speeds up stochastic Newton optimization.

For optimization on large-scale data, exactly calculating its solution may be computationally difficulty because of the large size of the data. In this paper we consider subsampled optimization for fast approximating the exact solution. In this approach, one gets a surrogate dataset by sampling from the full data, and …

Deep learning involves a difficult non-convex optimization problem with a large number of weights between any two adjacent layers of a deep structure. To handle large data sets or complicated networks, distributed training is needed, but the calculation of function, gradient, and Hessian is expensive. In particular, th…

New methods optimize functions faster with less gradient accuracy needed.

New averaging technique speeds up Newton method convergence.

A new training method uses multilevel minimization for machine learning.

A new method selects a representative subsample for efficient kernel density estimation.

A new model-free subsampling method using uniform designs is proposed.

A significant hurdle for analyzing large sample data is the lack of effective statistical computing and inference methods. An emerging powerful approach for analyzing large sample data is subsampling, by which one takes a random subsample from the original full sample and uses it as a surrogate for subsequent computati…

This paper optimizes subsampling for large datasets using Poisson distribution.

A new approach RA improves stochastic optimization by executing multiple steps between subsample updates.

Develops a method to optimize hyperparameters for subsampling methods.

A new method for optimization in probability space using Newton's flows.

A new method subsamples data without weights to improve model performance.

New research finds many coreset methods for logistic regression are not better than simple sampling.

Newton's method solves variational problems on manifolds.

Simple stochastic Newton and cubic Newton methods with fast convergence.

Newton methods improve CNN optimization, showing competitive accuracy.

A new method reduces variance in SGMCMC by preferentially subsampling data.

The rapid development of computing power and efficient Markov Chain Monte Carlo (MCMC) simulation algorithms have revolutionized Bayesian statistics, making it a highly practical inference method in applied work. However, MCMC algorithms tend to be computationally demanding, and are particularly slow for large datasets…

We generalize Newton-type methods for minimizing smooth functions to handle a sum of two convex functions: a smooth function and a nonsmooth function with a simple proximal mapping. We show that the resulting proximal Newton-type methods inherit the desirable convergence behavior of Newton-type methods for minimizing s…

We describe stochastic Newton and stochastic quasi-Newton approaches to efficiently solve large linear least-squares problems where the very large data sets present a significant computational burden (e.g., the size may exceed computer memory or data are collected in real-time). In our proposed framework, stochasticity…

Modified Newton step for online learning reduces matrix size for large datasets.

New equivalences found between subsampling and ridge regularization methods.

A new neural subsampling method reduces data volume for deep models.

Paper proposes an online covariance estimator for sketched Newton methods.

New Q-Newton's method avoids saddle points and converges quadratically.

The paper analyzes how subsampling Gaussian process regression works.

Differential privacy comes equipped with multiple analytical tools for the design of private data analyses. One important tool is the so-called "privacy amplification by subsampling" principle, which ensures that a differentially private mechanism run on a random subsample of a population provides higher privacy guaran…

Speeding up Markov Chain Monte Carlo (MCMC) for datasets with many observations by data subsampling has recently received considerable attention. A pseudo-marginal MCMC method is proposed that estimates the likelihood by data subsampling using a block-Poisson estimator. The estimator is a product of Poisson estimators,…

Unified approach to Bayesian inference with guarantees on covariance matrices.

Newton's method tackles nonlinear mappings into vector bundles with connections and retractions.

For massive data, the family of subsampling algorithms is popular to downsize the data volume and reduce computational burden. Existing studies focus on approximating the ordinary least squares estimate in linear regression, where statistical leverage scores are often used to define subsampling probabilities. In this p…

A new optimization method improves deep learning accuracy without hyper-parameter tuning.

This thesis disentangles Gauss-Newton and variational approximations in Bayesian deep learning.

For classification problems with significant class imbalance, subsampling can reduce computational costs at the price of inflated variance in estimating model parameters. We propose a method for subsampling efficiently for logistic regression by adjusting the class balance locally in feature space via an accept-reject …

New quasi-Newton method guarantees global superlinear convergence.

Approximate Newton methods are a standard optimization tool which aim to maintain the benefits of Newton's method, such as a fast rate of convergence, whilst alleviating its drawbacks, such as computationally expensive calculation or estimation of the inverse Hessian. In this work we investigate approximate Newton meth…

Improved training of large-scale neural networks with reduced variance noise.