Local Gradient Descent with local steps converges to the centralized model in the interpolation regime.
problem Understanding the implicit bias of Local Gradient Descent in the interpolation regime.
method Analyzing the implicit bias of Local Gradient Descent for classification tasks with linearly separable data.
result The aggregated global model from Local-GD converges exactly to the centralized model in the interpolation regime.
AutoStep MCMC adapts step size locally for better sampling efficiency.
problem Challenging step size selection for complex, multiscale targets.
method AutoStep MCMC uses a locally adaptive step size for involutive proposals.
result AutoStep MCMC is π-invariant, irreducible, and aperiodic.
GradSkip reduces local training steps for better communication efficiency.
problem High communication costs in distributed optimization.
method GradSkip redesigns ProxSkip to allow clients with less important data to take fewer local training steps.
result GradSkip converges linearly with reduced local training steps and same accelerated communication complexity.
LocalKMeans parallelizes Lloyd's algorithm for distributed data.
problem Efficiently clustering data across multiple machines.
method Parallel local iterations with synchronization every L steps.
result Higher required signal-to-noise ratio due to local steps.
New k-step policy gradient method avoids local optima in restricted policy classes.
problem Suboptimal local optima in policy gradient methods for restricted policy classes.
method Proposes a k-step policy gradient method to escape myopic local optima. result The method converges to near optimal solutions exponentially close to the optimal deterministic policy.
Proposes a graph dynamics prior for more accurate relational inference.
problem Identifying interactions in dynamical systems from observed dynamics.
method Graph Dynamics Prior (GDP) that uses error amplification in non-local polynomial filters.
result Reconstructs graphs more accurately than previous methods, robust to under-sampling.
This is a final step in a local classification of pseudo-Riemannian manifolds with parallel Weyl tensor that are not conformally flat or locally symmetric.
Efficient regularization mitigates catastrophic overfitting in single-step adversarial training.
problem Catastrophic overfitting in single-step adversarial training.
method ELLE regularization term to enforce local linearity of the loss function.
result Our regularization term effectively mitigates catastrophic overfitting without the drawbacks of previous methods.
New analysis shows Local SGD can achieve error scaling with only fixed number of communications.
problem Speeding up SGD by parallelizing across multiple workers with reduced communication overhead.
method Proposed and analyzed Local SGD method with a fixed number of communications independent of the number of steps.
result Achieves an error scaling as 1/(nT) with only a fixed number of communications (Ω(n)).
A new method automatically and dynamically sets learning rates in deep learning.
problem Determining the appropriate learning rate in deep learning tasks is challenging and often subjective.
method Local Quadratic Approximation (LQA) to automatically and dynamically set learning rates.
result The proposed method leads to nearly optimal learning rates in a computationally efficient way.
Proposes a method combining CNFs and rejection-resampling for sampling from unnormalized densities.
problem Sampling from unnormalized probability densities, especially multimodal ones.
method Combines continuous normalizing flows with rejection-resampling steps based on importance weights.
result The method improves sampling accuracy and performance compared to state-of-the-art methods.
Gradient descent converges linearly in finite-width networks with positive NTK and compatible conditions.
problem Local convergence of gradient descent in finite-width networks.
method Positive Neural Tangent Kernel (NTK), local Polyak-Łojasiewicz inequality, fixed-step containment in Locally Quasi-Convex Region (LQCR).
result Linear convergence achieved under specific conditions.
We investigate finite-time decoupled convergence in nonlinear two-time-scale stochastic approximation.
problem Achieving decoupled convergence in nonlinear two-time-scale stochastic approximation.
method Nested local linearity assumption, suitable step size selection, convergence analysis of matrix cross term, fourth-order moment convergence rates.
result Finite-time decoupled convergence rates can be achieved in nonlinear two-time-scale stochastic approximation with proper step size selection.
In this paper, we propose a simple, fast and easy to implement algorithm LOSSGRAD (locally optimal step-size in gradient descent), which automatically modifies the step-size in gradient descent during neural networks training. Given a function f, a point x, and the gradient ∇xf of f, we aim to find the s…
Distributed optimization often consists of two updating phases: local optimization and inter-node communication. Conventional approaches require working nodes to communicate with the server every one or few iterations to guarantee convergence. In this paper, we establish a completely different conclusion that each node…
Paper proposes faster method to find local minima in nonconvex optimization.
problem Escaping saddle points and finding local minima in nonconvex optimization.
method LENA (Last stEp shriNkAge) framework for faster perturbed stochastic gradient methods.
result LENA finds (ε,εH)-approximate local minima within ildeO(ε−3+εH−6) evaluations. Stochastic (sub)gradient methods require step size schedule tuning to perform well in practice. Classical tuning strategies decay the step size polynomially and lead to optimal sublinear rates on (strongly) convex problems. An alternative schedule, popular in nonconvex optimization, is called \emph{geometric step decay…
Paper develops Gaussian approximations and bootstrap for federated LSA with trade-off bounds.
problem Analyzing convergence rates and trade-offs in federated linear stochastic approximation.
method Established Berry-Esseen-type bounds for federated LSA, developed multiplier bootstrap for inference.
result First federated Gaussian approximations with explicit trade-off terms and non-asymptotic validity guarantees.
Inverted file and asymmetric distance computation (IVFADC) have been successfully applied to approximate nearest neighbor search and subsequently maximum inner product search. In such a framework, vector quantization is used for coarse partitioning while product quantization is used for quantizing residuals. In the ori…
GD converges in unstable regimes, even with oscillatory behavior.
problem Understanding convergence of GD in unstable regimes.
method Analysis of two-step gradient updates.
result Characterization of local conditions for convergence.
Multiple gossip steps improve decentralized optimization convergence.
problem Efficiently optimizing large-scale machine learning models with limited communication.
method Integrates multiple gossip steps between gradient descent iterations in compressed decentralized optimization.
result Convergence to within ε of the optimal value for smooth non-convex objectives.
funLOCI identifies clusters in functional data.
problem Identifying similar behavior in functional data.
method Divisive hierarchical clustering with additive model.
result funLOCI reduces the number of local clusters.
New optimization for federated learning with local models.
problem Training models with private data from multiple devices.
method Proposes a new optimization formulation and efficient SGD variants.
result Local steps can improve communication for heterogeneous data.
New techniques improve distributed training with compressed gradients.
problem Gradient mismatch problem in local error feedback.
method Step-ahead error feedback and error averaging techniques.
result Our methods handle gradient mismatch and train faster than full-precision training.
Proposes an exponentially increasing step-size for faster parameter estimation in statistical models.
problem Slow convergence of gradient descent in locally convex loss functions.
method Exponentially increasing step-size in gradient descent algorithm.
result Converges linearly to optimal solution under homogeneous assumptions.
We focus our attention on the notion of intrinsic Lipschitz graphs, inside a special class of metric spaces i.e. the Carnot groups. More precisely, we provide a characterization of locally intrinsic Lipschitz functions in Carnot groups of step 2 in terms of their intrinsic distributional gradients.
Data privacy is an important concern in learning, when datasets contain sensitive information about individuals. This paper considers consensus-based distributed optimization under data privacy constraints. Consensus-based optimization consists of a set of computational nodes arranged in a graph, each having a local ob…
FedSARSA converges with heterogeneous agents, achieving linear speed-up.
problem Convergence analysis of Federated SARSA with heterogeneous agents.
method Linear function approximation, local training, multi-step error expansion.
result FedSARSA achieves linear speed-up with respect to the number of agents.
ISALT uses inference to simulate SDEs with large time-steps, improving efficiency.
problem Efficiently simulating ergodic SDEs with large time-steps.
method Inference-based schemes adaptive to large time-steps (ISALT) from data.
result ISALT achieves significant time reduction and optimal accuracy.
Boosting Variational Inference improves posterior approximations with adaptive step-sizes.
problem Limited resources hinder the widespread adoption of Boosting Variational Inference.
method Characterized global curvature impact, introduced local curvature, and developed an approximate backtracking algorithm.
result New theoretical convergence rates and experimental validation demonstrate improved performance.
Asynchronous decentralized SGD with quantized and local updates converges in gossip model.
problem Scalable distributed machine learning challenges in decentralized optimization.
method Asynchronous decentralized SGD with quantization and local steps in gossip model.
result Decentralized optimization with quantization and local steps converges in asynchronous gossip model.
New adaptive step-size method for convex optimization without tuning.
problem Optimizing convex functions efficiently with stochastic gradients.
method Adapted Adaptive Gradient Descent Without Descent to stochastic setting.
result Stochastic gradient descent converges under various assumptions.
This paper is devoted to C2 a priori estimates for strictly locally convex radial graphs with prescribed Weingarten curvature and boundary in space forms. By constructing two-step continuity process and applying degree theory arguments, existence results in space forms are established for prescribed Gauss curvature …
We prove that a compact toric locally conformally Kähler manifold which is not Kähler admits a toric Vaisman structure, a fact which was conjectured in \cite{mmp}. This is the final step leading to the classification of compact toric locally conformally Kähler manifolds started in \cite{p} and \cite{mmp}. We also show,…
We take initial steps in studying PAC-MDP algorithms with limited adaptivity, that is, algorithms that change its exploration policy as infrequently as possible during regret minimization. This is motivated by the difficulty of running fully adaptive algorithms in real-world applications (such as medical domains), and …
Conditional gradients constitute a class of projection-free first-order algorithms for smooth convex optimization. As such, they are frequently used in solving smooth convex optimization problems over polytopes, for which the computational cost of orthogonal projections would be prohibitive. However, they do not enjoy …
This paper is the second part of a series of papers on noncommutative geometry and conformal geometry. In this paper, we compute explicitly the Connes-Chern character of an equivariant Dirac spectral triple. The formula that we obtain for which was used in the first paper of the series. The computation has two main ste…
Gradient method converges locally linearly for overparameterized Gaussian mixtures.
problem Learning Gaussian mixtures under overparameterization.
method Gradient-based method alternating short descent steps and long Polyak steps.
result Gradient method converges locally linearly to minimizers.
Gradient descent converges linearly for overparameterized linear networks.
problem Convergence of gradient descent for overparameterized neural networks.
method Local Polyak-Lojasiewicz and Descent Lemma for overparameterized linear models.
result Gradient descent achieves linear convergence for two-layer linear networks under relaxed assumptions.
ATLAS adapts HMC step size and trajectory length for complex geometries.
problem Sampling complex geometries with constant step size HMC/NUTS.
method Adapts step size and trajectory length using local Hessian and no U-turn condition.
result ATLAS accurately samples complex geometries, outperforming NUTS.
CW-EDMD improves prediction accuracy by learning local Koopman models for different state-space regions.
problem Inefficient global Koopman operator approximation for distinct local dynamics.
method Cluster-Weighted EDMD (CW-EDMD) learns a soft phase-space partition and per-cluster EDMD operators using EM objective.
result CW-EDMD significantly reduces prediction errors across various systems and configurations.
New method calibrates local volatility models to marginal distributions.
problem Calibrating local volatility models to specific marginal distributions.
method Inspired by volatility interpolation, constructs time-homogeneous or continuous local volatility functions.
result Efficient numerical algorithms for constructing local volatility functions.
We consider the non-parametric regression problem under Huber's ε-contamination model, in which an ε fraction of observations are subject to arbitrary adversarial noise. We first show that a simple local binning median step can effectively remove the adversary noise and this median estimator is minimax optimal up t…
In this paper, we obtain a localization formula in differential K-theory for S1-action. Then by combining an extension of Goette's result on the comparison of two types of equivariant η-invariants, we establish a version of localization formula for equivariant η-invariants. An important step of our approach is t…
Community detection in hypergraphs is explored. Under a generative hypergraph model called "d-wise hypergraph stochastic block model" (d-hSBM) which naturally extends the Stochastic Block Model from graphs to d-uniform hypergraphs, the asymptotic minimax mismatch ratio is characterized. For proving the achievability, w…
The family of Expectation-Maximization (EM) algorithms provides a general approach to fitting flexible models for large and complex data. The expectation (E) step of EM-type algorithms is time-consuming in massive data applications because it requires multiple passes through the full data. We address this problem by pr…
Deep learning model predicts traffic flows across entire network for multiple steps ahead.
problem Accurately forecasting future traffic flows across all network links.
method Spatial-Temporal Sequence to Sequence (STSeq2Seq) model combining seq2seq and graph convolution.
result STSeq2Seq achieves state-of-the-art performance in traffic forecasting.
Recent innovations in Information and Communication Technologies (ICT) provide new opportunities and challenges for integration of distributed energy resources (DERs) into the energy supply system as active market players. By increasing integration of DERs, novel market platform should be designed for these new market …