Stochastic Gradient Trees learn decision trees incrementally.
problem Learning decision trees using stochastic gradient information.
method Incremental learning setting, soft splits not used, new tree not constructed per update.
result Performs similarly to standard incremental classification trees, outperforms state of the art incremental regression trees, comparable to batch multi-instance learning methods.
Proposes a probabilistic optimization method for large-scale problems.
problem Large-scale regularized optimization problems.
method Develops a probabilistic interpretation of the incremental proximal gradient algorithm and uses Bayesian filtering.
result Makes it possible to solve large-scale problems using well-known Bayesian filters.
We analyze a fast incremental aggregated gradient method for optimizing nonconvex problems of the form min x ∑ i f i ( x ) \min_x \sum_i f_i(x) min x ∑ i f i ( x ) . Specifically, we analyze the SAGA algorithm within an Incremental First-order Oracle framework, and show that it converges to a stationary point provably faster than both gradient descent and s…
Paper improves convergence rates and step sizes for gradient algorithms.
problem Limited convergence rates and step sizes in existing algorithms.
method General proximal incremental aggregated gradient algorithm.
result Better and novel convergence results under general scheme.
CIAG method accelerates optimization for high-dimensional problems.
problem Training classifiers for high-dimensional problems with large datasets.
method CIAG method uses curvature information to accelerate IAG methods.
result Improved linear convergence rate over state-of-the-art IAG methods.
New model learns SDEs without gradient matching for non-uniform time increments.
problem Learning non-parametric drift and diffusion functions for SDEs.
method Formulates sensitivity equations for learning and optimizes path distributions.
result Robust and efficient learning of SDE systems with non-uniform time increments.
Paper tackles private incremental regression in streaming data.
problem Private incremental regression in streaming data.
method Generic transformation of private batch ERM to incremental ERM, and two mechanisms for private incremental regression.
result Improved empirical risk for certain regression problems.
Depth helps neural networks learn simpler solutions incrementally.
problem Understanding why deep neural networks generalize well despite complex architectures.
method Formal definition of incremental learning dynamics, theoretical analysis of depth and initialization effects, experiments with various models.
result Incremental learning dynamics can arise in deeper models, but not in shallow ones, under specific conditions.
GD with large init shows incremental learning in matrix factorization.
problem Understanding GD's behavior with large initial values in matrix factorization.
method Signal-to-noise ratio concepts and inductive arguments.
result Uncovering an incremental learning phenomenon in GD with large initialization.
Meta-learning approach improves object detection on new classes.
problem Deterioration of object detection performance on old classes in incremental settings.
method Meta-learning to reshape model gradients for optimal task adaptation.
result Meta-learning approach outperforms state-of-the-art methods in incremental object detection.
New method accelerates gradient optimization using curvature information.
problem Solving large-scale learning optimization problems efficiently.
method Curvature-aided gradient tracking step for incremental gradient optimization.
result Proves linear convergence rates for new methods.
GD learns matrix solutions incrementally, revealing insights into generalization.
problem Matrix sensing problem of recovering low-rank matrices from linear measurements.
method Fine-grained analysis of GD dynamics for matrix sensing.
result GD follows an incremental learning procedure, solving matrices of increasing ranks.
New method reduces complexity for nonconvex optimization problems.
problem Minimizing composite functions with random or finite sum inner mappings.
method Stochastic composite gradient method with incremental variance reduction.
result Achieves complexity similar to best first-order methods for expected-value and finite-sum nonconvex functions.
Efficiently adds or deletes data in GBDT models.
problem Traditional GBDT training requires all data to be accessed simultaneously, limiting add/delete operations.
method Proposes an online learning framework for GBDT supporting incremental and decremental learning.
result First work to unify incremental and decremental learning on GBDT in-place.
This paper improves convergence guarantees for SGD algorithms in non-convex smooth functions.
problem Theoretical convergence properties of SGD algorithms for non-convex smooth functions.
method Analysis of SGD algorithms with arbitrary data ordering for non-convex smooth functions.
result Enhanced convergence guarantees for incremental gradient and single shuffle SGD, improving the optimization term of convergence guarantee.
New algorithm improves on EM for streaming data, outperforming existing methods.
problem Processing high-volume, streaming data efficiently.
method Incremental stochastic Majorization-Minimization (MM) algorithm.
result The algorithm converges to a stationary point with vanishing gradient.
Optimal Volt/VAR control rules are designed using deep neural networks.
problem Designing optimal Volt/VAR control rules for distributed energy resources (DERs).
method Formulate optimal rule design as a bilevel program, then reformulate it as training a deep neural network (DNN). Use proximal gradient descent (PGD) iterations to emulate Volt/VAR dynamics.
result The proposed solution can be adapted to single/multi-phase feeders and achieves enhanced steady-state voltage profiles.
Many structured data-fitting applications require the solution of an optimization problem involving a sum over a potentially large number of measurements. Incremental gradient algorithms offer inexpensive iterations by sampling a subset of the terms in the sum. These methods can make great progress initially, but often…
Neural networks learn incrementally from orthogonal data, interpolating with minimal complexity.
problem Understanding the learning dynamics and implicit bias in ReLU networks with orthogonal data.
method Gradient flow analysis of two-layer ReLU networks from small initialization with orthogonal training data.
result The learned interpolator has a squared ℓ 2 \ell_2 ℓ 2 -norm scaling as n \sqrt{n} n , close to the minimal interpolator's complexity. A fast method for decentralized non-convex optimization over networks.
problem Decentralized non-convex optimization problems over a network of nodes.
method GT-SAGA, a randomized incremental gradient method that evaluates one component gradient per node per iteration.
result GT-SAGA achieves almost sure and mean-squared convergence to a first-order stationary point for general smooth non-convex problems.
Within a statistical learning setting, we propose and study an iterative regularization algorithm for least squares defined by an incremental gradient method. In particular, we show that, if all other parameters are fixed a priori, the number of passes over the data (epochs) acts as a regularization parameter, and prov…
TKIL improves class-balanced performance in incremental learning.
problem Catastrophic forgetting in sequential learning tasks.
method Introduces Tangent Kernel for Incremental Learning (TKIL) based on Neural Tangent Kernel (NTK).
result TKIL achieves better overall accuracy and variance across classes.
In this work we introduce a new optimisation method called SAGA in the spirit of SAG, SDCA, MISO and SVRG, a set of recently proposed incremental gradient algorithms with fast linear convergence rates. SAGA improves on the theory behind SAG and SVRG, with better theoretical convergence rates, and has support for compos…
SAGA achieves fast convergence on non-convex problems with RSC.
problem Non-convex optimization problems with restricted strong convexity.
method SAGA, a fast incremental gradient method, analyzed under RSC.
result SAGA achieves linear convergence up to statistical estimation accuracy under RSC.
Paper introduces an asynchronous optimization algorithm for parameter servers.
problem Solving optimization problems with asynchrony and general regularizers.
method Asynchronous incremental aggregated gradient algorithm in a parameter server framework.
result Established linear convergence rate and step-size choices for strong convex data loss.
A new method for machine learning updates reduces complexity and improves robustness.
problem Stochastic gradient updates are inefficient and sensitive to feature scaling.
method Incremental Gauss-Newton Descent (IGND) reduces the need for matrix operations and improves robustness.
result IGND improves robustness to sensitivity scaling and can be competitive with common stochastic optimizers.
Proposes an accelerated optimization algorithm for composite objectives.
problem Gradient-based optimization with sparse solutions and high dimensions.
method Inexact variable-metric proximal point algorithm (QNing) with limited-memory BFGS.
result Significant improvements over competing methods in training machine learning models.
This paper analyzes and improves EM algorithms for large datasets.
problem Scaling EM algorithm to large datasets.
method Incremental and stochastic EM algorithms, variance reduction.
result Established non-asymptotic convergence bounds for global convergence.
Proposes efficient model for continual learning that grows model over task-specific parameters.
problem Limited transfer learning ability and forgetting of earlier knowledge in existing methods.
method Filter and channel expansion method that grows model over previous task parameters.
result Better knowledge transfer and improved performance in task incremental learning.
A new learning algorithm improves neural network performance.
problem Improving performance of neural networks through better representations.
method Introducing crossprop, a new incremental learning algorithm based on meta-gradient descent.
result Crossprop learns and reuses feature representations, improving performance on new tasks.
New lower bounds for gradient methods in strongly convex finite-sum optimization.
problem Developing tight lower bounds for randomized gradient methods in finite-sum optimization.
method Deriving tight lower complexity bounds for SAG, SAGA, SVRG, SARAH, and related methods.
result Tight matches between lower bounds and upper bounds for various methods under specific conditions.
A new method for asynchronous eigenspace computation on the Grassmannian.
problem Asynchronous optimization for finite-sum eigenspace computation in distributed systems.
method Grassmannian incremental aggregation method that refreshes only arriving components and reuses cached gradients.
result Two-phase linear convergence with constants controlled by component spectral spreads.
An incremental/online state dynamic learning method is proposed for identification of the nonlinear Gaussian state space models. The method embeds the stochastic variational sparse Gaussian process as the probabilistic state dynamic model inside a particle filter framework. Model updating is done at measurement sample …
New method reduces inference variance for faster optimization.
problem High variance in black-box variational inference.
method Joint control variate addressing both data subsampling and Monte Carlo noise.
result Significantly reduced gradient variance, leading to faster optimization.
We propose HAMSI (Hessian Approximated Multiple Subsets Iteration), which is a provably convergent, second order incremental algorithm for solving large-scale partially separable optimization problems. The algorithm is based on a local quadratic approximation, and hence, allows incorporating curvature information to sp…
A new framework for training structured prediction models using smoothing.
problem Training smooth structured prediction models with non-smooth objectives.
method Smoothing over the maximum margin structured prediction objective to enable fast optimization.
result The proposed framework enables the use of efficient optimization algorithms for structured prediction.
New method tunes SMC samplers efficiently without high costs.
problem Tuning SMC samplers with unadjusted kernels is challenging.
method Greedy Incremental Divergence Minimization (GIDM) for step size tuning.
result GIDM reduces KL divergence and tunes SMC samplers efficiently.
This work presents GROUSE (Grassmanian Rank-One Update Subspace Estimation), an efficient online algorithm for tracking subspaces from highly incomplete observations. GROUSE requires only basic linear algebraic manipulations at each iteration, and each subspace update can be performed in linear time in the dimension of…
Paper explores using action-value gradients for policy improvement in off-policy actor-critic methods.
problem Improving policies using action-value gradients in off-policy stochastic actor-critic methods.
method Discusses and analyzes the use of action-value gradients for policy improvement, and proposes an incremental approach.
result Demonstrates the feasibility and incremental approach for following the policy gradient.
Paper evaluates different models for predicting credit default swap volatility.
problem Predicting the Implied Volatility of credit default swaps.
method SVM, Gradient Boosting, and Attention-GRU Hybrid model.
result Identifies strengths in classical and SOTA machine learning methods.
Paper investigates robustness to interference as a new training signal for meta-learning.
problem Improving incremental learning through robust representations.
method Directly minimizing catastrophic interference as a training signal.
result Representations learned to minimize interference lead to better incremental learning.
New algorithm estimates barycenters of distributions using Frank-Wolfe.
problem Estimating the average of arbitrary probability distributions.
method Frank-Wolfe optimization for Sinkhorn divergence, incrementally populating support.
result Converges in both discrete and continuous distributions, with proven rates.
We introduce incremental variational inference and apply it to latent Dirichlet allocation (LDA). Incremental variational inference is inspired by incremental EM and provides an alternative to stochastic variational inference. Incremental LDA can process massive document collections, does not require to set a learning …
We consider the problem of minimizing the sum of two convex functions: one is the average of a large number of smooth component functions, and the other is a general convex function that admits a simple proximal mapping. We assume the whole objective function is strongly convex. Such problems often arise in machine lea…
Efficiently merges multiple points to speed up BSGD SVM training.
problem Costly merging of points in BSGD SVM training.
method Merges more than two points at once to reduce training time.
result Significant speed-ups achieved without loss of accuracy.
A method to optimize deep networks by sequentially minimizing risk functions.
problem Optimizing deep networks during training to find global optima.
method Surfing: Iterative optimization over incrementally trained deep networks.
result The method can find global optima and improve compressed sensing performance.
A new algorithm for faster incremental kernel PCA and Nyström approximation.
problem Efficiency in handling streaming data and large datasets.
method Incremental kernel PCA and Nyström approximation using rank one updates.
result More computationally and memory efficient than existing methods.
Adaptive rule improves kernel-based gradient descent performance.
problem Improving convergence speed of kernel-based gradient descent algorithms.
method Empirical effective dimension for stopping rule, learning theory analysis, integral operator approach.
result Optimal learning rates and iteration bounds for KGD with adaptive stopping rule.