Paper improves convergence rates and step sizes for gradient algorithms.
problem Limited convergence rates and step sizes in existing algorithms.
method General proximal incremental aggregated gradient algorithm.
result Better and novel convergence results under general scheme.
CIAG method accelerates optimization for high-dimensional problems.
problem Training classifiers for high-dimensional problems with large datasets.
method CIAG method uses curvature information to accelerate IAG methods.
result Improved linear convergence rate over state-of-the-art IAG methods.
We analyze a fast incremental aggregated gradient method for optimizing nonconvex problems of the form min x ∑ i f i ( x ) \min_x \sum_i f_i(x) min x ∑ i f i ( x ) . Specifically, we analyze the SAGA algorithm within an Incremental First-order Oracle framework, and show that it converges to a stationary point provably faster than both gradient descent and s…
Paper introduces an asynchronous optimization algorithm for parameter servers.
problem Solving optimization problems with asynchrony and general regularizers.
method Asynchronous incremental aggregated gradient algorithm in a parameter server framework.
result Established linear convergence rate and step-size choices for strong convex data loss.
A new method for asynchronous eigenspace computation on the Grassmannian.
problem Asynchronous optimization for finite-sum eigenspace computation in distributed systems.
method Grassmannian incremental aggregation method that refreshes only arriving components and reuses cached gradients.
result Two-phase linear convergence with constants controlled by component spectral spreads.
New method accelerates gradient optimization using curvature information.
problem Solving large-scale learning optimization problems efficiently.
method Curvature-aided gradient tracking step for incremental gradient optimization.
result Proves linear convergence rates for new methods.
A new algorithm estimates aggregate marginals from noisy data in an online manner.
problem Estimating aggregate marginals of a Markov chain from noisy aggregate observations.
method Sliding window Sinkhorn belief propagation (SW-SBP) algorithm.
result Demonstrated improved performance on inferring population flow.
AANets balance stability and plasticity in CIL.
problem Stability-plasticity dilemma in class-incremental learning.
method Adaptive Aggregation Networks (AANets) with stable and plastic residual blocks.
result AANets improve performance on CIL benchmarks.
Stochastic Gradient Trees learn decision trees incrementally.
problem Learning decision trees using stochastic gradient information.
method Incremental learning setting, soft splits not used, new tree not constructed per update.
result Performs similarly to standard incremental classification trees, outperforms state of the art incremental regression trees, comparable to batch multi-instance learning methods.
BiLA uses variational Bayesian inference to aggregate noisy labels online.
problem Aggregating noisy labels from crowd workers in real-time.
method Variational Bayesian inference and stochastic optimization.
result BiLA reduces label error by at least 10-1.5% points.
Proposes a probabilistic optimization method for large-scale problems.
problem Large-scale regularized optimization problems.
method Develops a probabilistic interpretation of the incremental proximal gradient algorithm and uses Bayesian filtering.
result Makes it possible to solve large-scale problems using well-known Bayesian filters.
New model learns SDEs without gradient matching for non-uniform time increments.
problem Learning non-parametric drift and diffusion functions for SDEs.
method Formulates sensitivity equations for learning and optimizes path distributions.
result Robust and efficient learning of SDE systems with non-uniform time increments.
Paper tackles private incremental regression in streaming data.
problem Private incremental regression in streaming data.
method Generic transformation of private batch ERM to incremental ERM, and two mechanisms for private incremental regression.
result Improved empirical risk for certain regression problems.
Depth helps neural networks learn simpler solutions incrementally.
problem Understanding why deep neural networks generalize well despite complex architectures.
method Formal definition of incremental learning dynamics, theoretical analysis of depth and initialization effects, experiments with various models.
result Incremental learning dynamics can arise in deeper models, but not in shallow ones, under specific conditions.
GD with large init shows incremental learning in matrix factorization.
problem Understanding GD's behavior with large initial values in matrix factorization.
method Signal-to-noise ratio concepts and inductive arguments.
result Uncovering an incremental learning phenomenon in GD with large initialization.
Media tone around earnings announcements predicts stock returns.
problem Determining if media tone around earnings announcements provides useful information for stock prices.
method Conducted an event study on media tone around earnings announcements for nonfinancial S&P 500 firms.
result Media tone around earnings announcements predicts abnormal stock returns.
Meta-learning approach improves object detection on new classes.
problem Deterioration of object detection performance on old classes in incremental settings.
method Meta-learning to reshape model gradients for optimal task adaptation.
result Meta-learning approach outperforms state-of-the-art methods in incremental object detection.
GD learns matrix solutions incrementally, revealing insights into generalization.
problem Matrix sensing problem of recovering low-rank matrices from linear measurements.
method Fine-grained analysis of GD dynamics for matrix sensing.
result GD follows an incremental learning procedure, solving matrices of increasing ranks.
New method reduces complexity for nonconvex optimization problems.
problem Minimizing composite functions with random or finite sum inner mappings.
method Stochastic composite gradient method with incremental variance reduction.
result Achieves complexity similar to best first-order methods for expected-value and finite-sum nonconvex functions.
Paper improves privacy-preserving measurement of advertising incrementality.
problem Privacy degradation in randomized lift tests for advertising measurement.
method Formulates a robust causal decision problem under signal losses, projecting clean worlds onto incrementality.
result Sharp decision frontier shows valid certification or rejection outside the frontier.
Efficiently adds or deletes data in GBDT models.
problem Traditional GBDT training requires all data to be accessed simultaneously, limiting add/delete operations.
method Proposes an online learning framework for GBDT supporting incremental and decremental learning.
result First work to unify incremental and decremental learning on GBDT in-place.
This paper improves convergence guarantees for SGD algorithms in non-convex smooth functions.
problem Theoretical convergence properties of SGD algorithms for non-convex smooth functions.
method Analysis of SGD algorithms with arbitrary data ordering for non-convex smooth functions.
result Enhanced convergence guarantees for incremental gradient and single shuffle SGD, improving the optimization term of convergence guarantee.
New algorithm improves on EM for streaming data, outperforming existing methods.
problem Processing high-volume, streaming data efficiently.
method Incremental stochastic Majorization-Minimization (MM) algorithm.
result The algorithm converges to a stationary point with vanishing gradient.
GradiVeQ reduces CNN training time by 50% with 5X faster gradient aggregation.
problem Significant communication costs in gradient aggregation for distributed CNN training.
method GradiVeQ uses PCA to vector quantize gradients for direct RAR aggregation.
result GradiVeQ reduces wall-clock gradient aggregation time by more than 5X.
Securely aggregates user-held gradients in federated learning without revealing individual data.
problem Securely aggregating gradients from users in federated learning without exposing individual data.
method Novel communication-efficient Secure Aggregation protocol for high-dimensional data, tolerating up to 1/3 users failing.
result Offers significant communication efficiency for federated learning with high-dimensional data.
Optimal Volt/VAR control rules are designed using deep neural networks.
problem Designing optimal Volt/VAR control rules for distributed energy resources (DERs).
method Formulate optimal rule design as a bilevel program, then reformulate it as training a deep neural network (DNN). Use proximal gradient descent (PGD) iterations to emulate Volt/VAR dynamics.
result The proposed solution can be adapted to single/multi-phase feeders and achieves enhanced steady-state voltage profiles.
Many structured data-fitting applications require the solution of an optimization problem involving a sum over a potentially large number of measurements. Incremental gradient algorithms offer inexpensive iterations by sampling a subset of the terms in the sum. These methods can make great progress initially, but often…
Paper studies fundamental limits of communication in distributed learning.
problem Communication efficiency in model aggregation for distributed learning.
method Rate-Distortion approach to model aggregation as a vector Gaussian CEO problem.
result Derives rate region bound and sum-rate-distortion function for model aggregation.
Neural networks learn incrementally from orthogonal data, interpolating with minimal complexity.
problem Understanding the learning dynamics and implicit bias in ReLU networks with orthogonal data.
method Gradient flow analysis of two-layer ReLU networks from small initialization with orthogonal training data.
result The learned interpolator has a squared ℓ 2 \ell_2 ℓ 2 -norm scaling as n \sqrt{n} n , close to the minimal interpolator's complexity. A fast method for decentralized non-convex optimization over networks.
problem Decentralized non-convex optimization problems over a network of nodes.
method GT-SAGA, a randomized incremental gradient method that evaluates one component gradient per node per iteration.
result GT-SAGA achieves almost sure and mean-squared convergence to a first-order stationary point for general smooth non-convex problems.
Within a statistical learning setting, we propose and study an iterative regularization algorithm for least squares defined by an incremental gradient method. In particular, we show that, if all other parameters are fixed a priori, the number of passes over the data (epochs) acts as a regularization parameter, and prov…
TKIL improves class-balanced performance in incremental learning.
problem Catastrophic forgetting in sequential learning tasks.
method Introduces Tangent Kernel for Incremental Learning (TKIL) based on Neural Tangent Kernel (NTK).
result TKIL achieves better overall accuracy and variance across classes.
In this work we introduce a new optimisation method called SAGA in the spirit of SAG, SDCA, MISO and SVRG, a set of recently proposed incremental gradient algorithms with fast linear convergence rates. SAGA improves on the theory behind SAG and SVRG, with better theoretical convergence rates, and has support for compos…
Non-affine aggregation rules cannot preserve monotonicity in convex learning.
problem Designing non-affine aggregation rules that maintain monotonicity in convex learning.
method Proving that monotonicity of aggregated gradients is preserved only if the aggregation rule is positively affine.
result Non-affine aggregation prevents steady convergence and substantially degrades algorithmic stability.
A new method reduces communication in distributed learning by skipping less informative gradient updates.
problem Efficient communication in distributed machine learning.
method Quantizes and skips less informative gradients to reduce communication overhead.
result Proves linear convergence rate similar to gradient descent with significant communication savings.
Monotone aggregation of dependent random vectors has an absolutely continuous distribution under certain conditions.
problem Monotone aggregation of dependent random vectors
method Coordinatewise monotonicity and uniform lower-increment conditions
result One-dimensional push-forwards of dependent random vectors have an absolutely continuous distribution
Federated learning improves by unbiased gradient aggregation and controllable meta updating.
problem Gradient biases and inconsistency between target and optimization objectives in federated averaging.
method Unbiased gradient aggregation with keep-trace gradient descent and gradient evaluation strategy, controllable meta updating with small data samples.
result Faster convergence and higher accuracy with different network architectures in various FL settings.
Policy gradient methods with aggregated states can achieve better performance than approximate policy iteration.
problem Approximation errors in policy and value function approximations.
method State-aggregated representations and policy gradient methods.
result Policy gradient methods can achieve a per-period regret bounded by ε, while approximate policy iteration and value iteration have a higher regret.
Skewness dispersion predicts future stock market returns, especially in months with monetary policy announcements.
problem Predicting future stock market returns using skewness dispersion.
method Cross-sectional analysis of firm-level realized skewness and stock market returns.
result Skewness dispersion is a significant predictor of future stock market returns, robust to various estimation methods.
SAGA achieves fast convergence on non-convex problems with RSC.
problem Non-convex optimization problems with restricted strong convexity.
method SAGA, a fast incremental gradient method, analyzed under RSC.
result SAGA achieves linear convergence up to statistical estimation accuracy under RSC.
A new method for machine learning updates reduces complexity and improves robustness.
problem Stochastic gradient updates are inefficient and sensitive to feature scaling.
method Incremental Gauss-Newton Descent (IGND) reduces the need for matrix operations and improves robustness.
result IGND improves robustness to sensitivity scaling and can be competitive with common stochastic optimizers.
CodedReduce combines tree topology and gradient coding for efficient and resilient gradient aggregation.
problem Efficient and robust gradient aggregation in distributed learning.
method CodedReduce combines tree topology and gradient coding to overcome bandwidth bottlenecks and straggler delays.
result CodedReduce achieves up to 27.2x speedup over benchmarks GC and RAR.
LASG improves communication efficiency in distributed learning.
problem Efficiently communicating gradients in distributed machine learning.
method Develops a new stochastic gradient descent approach, LASG, that predicts and selects significant communication rounds.
result Achieves communication savings by an order of magnitude in federated learning.
This paper analyzes and improves EM algorithms for large datasets.
problem Scaling EM algorithm to large datasets.
method Incremental and stochastic EM algorithms, variance reduction.
result Established non-asymptotic convergence bounds for global convergence.
We propose an inexact variable-metric proximal point algorithm to accelerate gradient-based optimization algorithms. The proposed scheme, called QNing can be notably applied to incremental first-order methods such as the stochastic variance-reduced gradient descent algorithm (SVRG) and other randomized incremental opti…
G-PATE generates private data with high utility using teacher-discriminator aggregation.
problem Privacy concerns in large-scale data sharing for machine learning.
method Generative adversarial nets combined with private gradient aggregation among discriminators.
result Significantly improves privacy budget efficiency and data utility.
Phocas improves robustness in SGD against Byzantine failures.
problem Byzantine failures in distributed SGD.
method Proposes a novel aggregation rule for Byzantine-resilient SGD.
result Proves Byzantine resilience of the proposed aggregation rules.
AdaBatch speeds up SGD for sparse problems with minimal code changes.
problem Efficiently aggregating gradients in stochastic optimization for sparse problems.
method AdaBatch: a new aggregation operator for mini-batches.
result AdaBatch achieves similar sample efficiency to standard SGD but with larger batch sizes.