arXiv research

New method reduces memory usage in deep HRNNs by replacing gradient backpropagation with local losses.

We consider distributed convex optimization problems originated from sample average approximation of stochastic optimization, or empirical risk minimization in machine learning. We assume that each machine in the distributed computing system has access to a local empirical loss function, constructed with i.i.d. data sa…

New insights on pruning deep networks by preserving function locality.

Q-SHAP efficiently calculates feature contributions in boosting trees.

New score matching method estimates local intrinsic dimension efficiently.

The aim of this paper is to provide new theoretical and computational understanding on two loss regularizations employed in deep learning, known as local entropy and heat regularization. For both regularized losses we introduce variational characterizations that naturally suggest a two-step scheme for their optimizatio…

New algorithms achieve optimal DP convex optimization with linear time and gradient computations.

A new method uses local sensitivity to improve importance sampling for approximating complex loss functions.

It is well-known from the work of Schönbucher (2005) that the marginal laws of a loss process can be matched by a unit increasing time inhomogeneous Markov process, whose deterministic jump intensity is called local intensity. The Stochastic Local Intensity (SLI) models such as the one proposed by Arnsdorf and Halperin…

Develops LSH schemes for f-divergences and mutual information loss.

This paper proposes an alternative to E2E training for deep networks, reducing memory footprint.

Forward gradients improve neural network training without backpropagation issues.

We propose a new regularization method based on virtual adversarial loss: a new measure of local smoothness of the conditional label distribution given input. Virtual adversarial loss is defined as the robustness of the conditional label distribution around each input data point against local perturbation. Unlike adver…

Local SGD with periodic averaging achieves faster convergence with less communication.

Gaussian process regression loses locality in high dimensions, affecting molecular energy surface fitting.

Polyak step size GD reaches final radius of convergence after log iterations.

The L1 loss landscape of neural nets near local minima behaves differently, revealing exponential decay and increased vertex density.

There are two important statements regarding the Trautman-Bondi mass at null infinity: one is the positivity, and the other is the Bondi mass loss formula, which are both global in nature. In this note, we compute the limit of the Wang-Yau quasi-local mass on unit spheres at null infinity of an asymptotically flat spac…

Recent work has noted that all bad local minima can be removed from neural network loss landscapes, by adding a single unit with a particular parameterization. We show that the core technique from these papers can be used to remove all bad local minima from any loss landscape, so long as the global minimum has a loss o…

We use barcodes to analyze neural networks' loss surfaces, revealing important properties.

Proposes ContSup to boost local learning by supplying context between isolated modules.

Researchers improve visualization of neural network loss landscapes.

Optimizing proper loss yields calibrated models under specific conditions.

FeDXL tackles federated learning for X-risk optimization.

Paper addresses ERM in LDP, reducing sample complexity for smooth and convex losses.

This research develops a new model for cyber risk and insurance pricing.

Develops a deep learning method for enforcing no-arbitrage in local volatility surfaces.

Local update methods' performance depends on learning rates, affecting convergence rates and alignment with true loss.

Local asymptotic minimax risk bounds in a locally asymptotically mixture of normal family of distributions have been investigated under asymmetric loss functions and the asymptotic distribution of the optimal estimator that attains the bound has been obtained.

L2G2G improves graph autoencoder accuracy without sacrificing scalability.

Proposes an exponentially increasing step-size for faster parameter estimation in statistical models.

Supervised training of neural networks for classification is typically performed with a global loss function. The loss function provides a gradient for the output layer, and this gradient is back-propagated to hidden layers to dictate an update direction for the weights. An alternative approach is to train the network …

Local Gradient Descent with local steps converges to the centralized model in the interpolation regime.

Study reveals properties of local minima in ReLU networks.

Proposes a new contrastive loss for semi-supervised medical image segmentation.

Quantification of the stationary points and the associated basins of attraction of neural network loss surfaces is an important step towards a better understanding of neural network loss surfaces at large. This work proposes a novel method to visualise basins of attraction together with the associated stationary points…

There is growing interest in large-scale machine learning and optimization over decentralized networks, e.g. in the context of multi-agent learning and federated learning. Due to the imminent need to alleviate the communication burden, the investigation of communication-efficient distributed optimization algorithms - p…

Optimizes exp-concave losses with a new risk bound.

We identify a class of over-parameterized deep neural networks with standard activation functions and cross-entropy loss which provably have no bad local valley, in the sense that from any point in parameter space there exists a continuous path on which the cross-entropy loss is non-increasing and gets arbitrarily clos…

Hill-ADAM optimizes loss landscapes by exploring state space deterministically.

In deep learning, \textit{depth}, as well as \textit{nonlinearity}, create non-convex loss surfaces. Then, does depth alone create bad local minima? In this paper, we prove that without nonlinearity, depth alone does not create bad local minima, although it induces non-convex loss surface. Using this insight, we greatl…

Alternating direction method of multiplier (ADMM) is a popular method used to design distributed versions of a machine learning algorithm, whereby local computations are performed on local data with the output exchanged among neighbors in an iterative fashion. During this iterative process the leakage of data privacy a…

Piecewise linear activations create many spurious local minima in neural networks.

We study distributed algorithms for expected loss minimization where the datasets are large and have to be stored on different machines. Often we deal with minimizing the average of a set of convex functions where each function is the empirical risk of the corresponding part of the data. In the distributed setting wher…

Deep ReLU networks with extra parameters have mostly good loss landscapes.

Study reveals sharp characterisation of local minima in neural network loss landscapes.

A new method for high-dimensional classification using Bernstein polynomials.

CurvSSL improves SSL by aligning local manifold curvature.