arXiv research

A new energy-efficient pruning method for federated learning.

RSO uses random weight perturbations to train deep networks without gradients.

The rise of deep learning in recent years has brought with it increasingly clever optimization methods to deal with complex, non-linear loss functions. These methods are often designed with convex optimization in mind, but have been shown to work well in practice even for the highly non-convex optimization associated w…

The ADAM optimizer is exceedingly popular in the deep learning community. Often it works very well, sometimes it doesn't. Why? We interpret ADAM as a combination of two aspects: for each weight, the update direction is determined by the sign of stochastic gradients, whereas the update magnitude is determined by an esti…

This work analyzes the statistical properties of adaptive gradient methods.

New method uses randomized sparse neural networks to solve time-dependent PDEs more accurately and efficiently.

FedCONST adapts update magnitudes to enhance feature generalization in FL.

Synthetic data replaces model updates in federated learning.

A new pruning criterion reduces model size and improves performance.

We propose and analyze a new parallel coordinate descent method---`NSync---in which at each iteration a random subset of coordinates is updated, in parallel, allowing for the subsets to be chosen non-uniformly. We derive convergence rates under a strong convexity assumption, and comment on how to assign probabilities t…

Analyze SGD with biased gradients, improving convergence rates and accuracy.

PrIU optimizes machine learning model updates after data cleaning.

DG separates successes and failures by gating updates with advantage and surprisal.

Distributed training of massive machine learning models, in particular deep neural networks, via Stochastic Gradient Descent (SGD) is becoming commonplace. Several families of communication-reduction methods, such as quantization, large-batch methods, and gradient sparsification, have been proposed. To date, gradient s…

Most learning algorithms are not invariant to the scale of the function that is being approximated. We propose to adaptively normalize the targets used in learning. This is useful in value-based reinforcement learning, where the magnitude of appropriate value approximations can change over time when we update the polic…

Logit dynamics formula reveals self-regulation in softmax policy gradient methods.

Develops methods to measure and set function-space learning rates in neural networks.

Tensor networks and RNNs are equivalent, improving wave function encoding.

Efficiently optimizes boolean functions using multilinear polynomials and exponential weight updates.

We propose a scalable stochastic variational approach to GP classification building on Polya-Gamma data augmentation and inducing points. Unlike former approaches, we obtain closed-form updates based on natural gradients that lead to efficient optimization. We evaluate the algorithm on real-world datasets containing up…

In this paper, we propose a method for training neural networks when we have a large set of data with weak labels and a small amount of data with true labels. In our proposed model, we train two neural networks: a target network, the learner and a confidence network, the meta-learner. The target network is optimized to…

BICompFL tackles bi-directional compression challenges in stochastic FL, reducing communication costs by an order of magnitude.

DaRE forests enable efficient data deletion from random forests.

Mathematical methods characterize RNNs' asymptotics as hidden units and data grow.

Deep reinforcement learning methods attain super-human performance in a wide range of environments. Such methods are grossly inefficient, often taking orders of magnitudes more data than humans to achieve reasonable performance. We propose Neural Episodic Control: a deep reinforcement learning agent that is able to rap…

We present a novel Metropolis-Hastings method for large datasets that uses small expected-size minibatches of data. Previous work on reducing the cost of Metropolis-Hastings tests yield variable data consumed per sample, with only constant factor reductions versus using the full dataset for each sample. Here we present…

A new sampler for complex discrete distributions efficiently updates all variables in parallel.

Unified learning-rate scale for CNNs and ResNets, avoiding depth imbalance.

System learns optimizer hyperparameters to generalize across tasks.

The $L_1$-regularized models are widely used for sparse regression or classification tasks. In this paper, we propose the orthant-wise passive descent algorithm (OPDA) for optimizing $L_1$-regularized models, as an improved substitute of proximal algorithms, which are the standard tools for optimizing the models nowada…

We propose a simple yet effective technique to simplify the training and the resulting model of neural networks. In back propagation, only a small subset of the full gradient is computed to update the model parameters. The gradient vectors are sparsified in such a way that only the top-k elements (in terms of magnitude…

Policy gradient algorithms typically combine discounted future rewards with an estimated value function, to compute the direction and magnitude of parameter updates. However, for most Reinforcement Learning tasks, humans can provide additional insight to constrain the policy learning. We introduce a general method to i…

NS-GAN mode collapse due to sample weighting inversion, solved with MM-nsat.

Distributed model training is vulnerable to byzantine system failures and adversarial compute nodes, i.e., nodes that use malicious updates to corrupt the global model stored at a parameter server (PS). To guarantee some form of robustness, recent work suggests using variants of the geometric median as an aggregation r…

Improved diffusion models for image synthesis with better training dynamics.

Paper proves convergence for private FL on non-Lipschitz convex objectives using normalization instead of clipping.

Magnitude homology reveals that graphs can have torsion subgroups.

Deep neural networks have gained tremendous popularity in last few years. They have been applied for the task of classification in almost every domain. Despite the success, deep networks can be incredibly slow to train for even moderate sized models on sufficiently large datasets. Additionally, these networks require l…

New method speeds up inference for non-conjugate Gaussian processes.

Study separates learning rate effects from adaptive gradient methods.

Effective regularisation of neural networks is essential to combat overfitting due to the large number of parameters involved. We present an empirical analogue to the Lipschitz constant of a feed-forward neural network, which we refer to as the maximum gain. We hypothesise that constraining the gain of a network will h…

Defines magnitude for length spaces with measures, agreeing with finite spaces' magnitude.

SparseProp speeds up SNN simulations and training by four orders of magnitude.

The paper provides convergence guarantees for multicalibration gradient boosting.

This research quantifies neural networks using magnitude, a topological invariant.

Magnitude of manifolds linked to Riesz energies and beta functions.

Modern classification problems frequently present mild to severe label imbalance as well as specific requirements on classification characteristics, and require optimizing performance measures that are non-decomposable over the dataset, such as F-measure. Such measures have spurred much interest and pose specific chall…

Lookahead pruning extends single-layer optimization to multi-layer, outperforming magnitude-based pruning.