Gradient method converges locally linearly for overparameterized Gaussian mixtures.
problem Learning Gaussian mixtures under overparameterization.
method Gradient-based method alternating short descent steps and long Polyak steps.
result Gradient method converges locally linearly to minimizers.
Gradient descent can be very slow in escaping saddle points.
problem Gradient descent's slowness in escaping saddle points.
method Gradient descent with natural random initialization and non-pathological functions.
result Gradient descent can take exponential time to escape saddle points.
Optimal Liouville theorem for minimal disks in any codimension.
problem Characterizing harmonic functions on minimal disks in high-dimensional spaces.
method Analyzing harmonic functions and using Liouville's theorem.
result Optimal Liouville theorem for minimal disks in any codimension.
A new gradient descent method speeds up in flat regions and slows in steep directions.
problem Improving the speed and stability of gradient descent algorithms.
method Introducing a 'power gradient' where each gradient component is replaced by its H-th power, with 0<H<1. result The new gradient descent methods achieve significantly better performances, especially for Nesterov accelerated gradient and AMSGrad.
Study of manifolds with nonnegative Ricci curvature and slow relative volume growth.
problem Understanding fundamental groups of manifolds with specific volume growth.
method Defined a function RV(s) to describe volume growth and studied fundamental groups with slow relative volume growth.
result If RV(s) grows sublinearly, fundamental groups are almost abelian or finite.
Analysis of a stochastic system showing convergence to an averaged model with Gaussian deviations.
problem Convergence analysis of a perturbed compositional gradient flow system.
method Separation of scales and averaging principle applied to stochastic differential equations.
result The slow motion of the system can be approximated by a standard perturbed gradient flow or SCGD algorithm.
Online learning algorithms have impressive convergence properties when it comes to risk minimization and convex games on very large problems. However, they are inherently sequential in their design which prevents them from taking advantage of modern multi-core architectures. In this paper we prove that online learning …
Researchers provide high-order approximations of slow invariant manifolds for atmospheric models.
problem Constructing slow invariant manifolds for atmospheric models with high accuracy.
method Flow Curvature Method
result Eighteenth-order approximation of the slow manifold for generalized model, thirteenth-order for conservative model.
A new travel time tomography method uses adaptive dictionaries to model slowness variations.
problem Modeling and reconstructing slowness maps with varying scales and discontinuities.
method Local model (sparse patches) and global model (smooth constraints) integrated into a maximum a posteriori formulation.
result The LST approach effectively models both smooth and discontinuous slowness features.
This work interprets SFA through variational inference, relaxing linearity constraints.
problem Recover non-linear SFA from variational inference.
method Probabilistic interpretation of SFA through variational inference, relaxing linearity constraints.
result Reinterprets SFA as a variational framework, allowing slowness as a regularizer to reconstruction loss.
Gradient-based method extracts slow features from high-dimensional data.
problem Extracting meaningful low-dimensional features from high-dimensional, temporally varying data.
method Power Slow Feature Analysis (PowerSFA) using gradient-based training of differentiable architectures.
result PowerSFA effectively extracts meaningful low-dimensional features in various data types.
TAEs discover slow modes but mix them with max variance modes.
problem Discovering slow modes in dynamical systems.
method Theoretical and numerical analysis of TAEs.
result TAEs learn a mixture of slow and max variance modes.
This work proposes a geometric approach to identify slow invariant manifolds in dynamical systems.
problem Identifying slow invariant manifolds in multiple time-scale dynamical systems.
method Differential geometric concepts for submanifolds, sectional curvature, flow invariance.
result Necessary condition for slow invariant manifold invariance stated in terms of differential geometry.
Method learns dynamics of slow variables from stochastic data.
problem Modeling unknown multiscale stochastic systems with limited data.
method Data-driven approach to learn effective dynamics from bursts of observation data.
result Generative model accurately captures effective dynamics of slow variables.
For the problem of high-dimensional sparse linear regression, it is known that an ℓ0-based estimator can achieve a 1/n "fast" rate on the prediction error without any conditions on the design matrix, whereas in absence of restrictive conditions on the design matrix, popular polynomial-time methods only guarante…
New framework for distributed nonparametric estimation under slow communication.
problem Efficiently estimate nonparametric models across multiple nodes with limited communication.
method Developed a general framework for nonparametric estimation under communication constraints.
result Derived minimax lower and upper bounds for various models.
We consider in a market model the cooperative emergence of value due to a positive feedback between perception of needs and demand. Here we consider also a negative feedback from production of the traded products, and find that this cooperativity is robust, provided that the production rate is slow. Cooperativity is fo…
New geometric approach to slow invariant manifolds in dynamical systems.
problem Characterizing slow invariant manifolds in a coordinate-independent manner.
method Exploiting curvature concepts and variational approach in Hamiltonian mechanics.
result Differential geometric definition of slow invariant manifolds proposed.
We provide a rigorous numerical computation method to validate periodic, homoclinic and heteroclinic orbits as the continuation of singular limit orbits for the fast-slow system x′=f(x,y,ε),y′=εg(x,y,ε) with one-dimensional slow variable y. Our validation procedure is based on topological tools called isolatin…
Bayesian optimization (BO) aims to minimize a given blackbox function using a model that is updated whenever new evidence about the function becomes available. Here, we address the problem of BO under partially right-censored response data, where in some evaluations we only obtain a lower bound on the function value. T…
In many domains, scientists build complex simulators of natural phenomena that encode their hypotheses about the underlying processes. These simulators can be deterministic or stochastic, fast or slow, constrained or unconstrained, and so on. Optimizing the simulators with respect to a set of parameter values is common…
Direct proof shows adaptive gradient descent converges near-linearly for convex functions.
problem Proving near-linear convergence of adaptive gradient descent for convex functions.
method Direct Lyapunov-based argument for convex functions with unique minimizer.
result Direct proof of near-linear convergence for convex functions.
Regularization leads to balancedness in deep linear networks.
problem Balancedness in deep linear networks.
method Geometric invariant theory and Riemannian geometry of fibers.
result Balancing flows converge to the balanced manifold at a uniform exponential rate.
Derives a biologically plausible neural network for Slow Feature Analysis.
problem Learning latent features from time series data.
method Starting from an SFA objective, derives Bio-SFA with a biologically plausible neural network implementation.
result Validates Bio-SFA on naturalistic stimuli, reproducing interesting properties of brain cells.
New bounds show polyhedral surrogates are optimal for generalization.
problem Proving generalization rates for polyhedral loss functions.
method Developed two general results for polyhedral surrogates.
result Polyhedral surrogates provide linear surrogate regret bounds, translating directly to target rates.
Paper introduces slow kill for efficient large-scale variable screening.
problem Challenges in variable selection and parameter estimation for big data.
method Nonconvex constrained optimization, adaptive \(\ell_2\)-shrinkage, and increasing learning rates.
result Slow kill outperforms state-of-the-art algorithms in various situations.
Fast algorithm tackles nonsmooth optimization problems.
problem Nonsmooth optimization in risk minimization.
method Continuation algorithm for nonsmooth regularized risk minimization.
result Achieves fastest known rates of convergence for strongly convex and general convex problems.
Develops variational inference for Neyman-Scott processes for faster sampling.
problem Slow mixing time in MCMC for posterior sampling in Neyman-Scott processes.
method Variational inference algorithm for Neyman-Scott processes, minimizing KL divergence.
result Achieves better prediction performance than MCMC with limited computational time.
Gradient descent converges to perfect classification in neural nets for non-separable data.
problem Classifying linearly non-separable data using neural networks.
method Analysis of gradient descent dynamics in neural networks with sufficient but not large number of neurons.
result Gradient descent converges to global minima with perfect classification in the landscape of minimization problems.
New algorithms solve large-scale rank minimization problems efficiently.
problem Large-scale rank minimization problems.
method Define and apply bi-trace and tri-trace norms to rank minimization problems; design efficient linearized alternating minimization algorithms.
result Proved algorithms converge to critical points; provide RSC and MC error bounds.
Study the averaging principle for non-autonomous slow-fast systems and apply it to financial local stochastic volatility models.
problem Understanding the behavior of non-autonomous slow-fast systems of stochastic differential equations.
method Prove the averaging principle under specific conditions and apply it to a financial model.
result Prices of derivatives converge to those calculated using the limit model under a risk-neutral measure.
A new geometric approach to identify slow invariant manifolds in complex systems.
problem The mathematical definition of slow invariant manifolds is unsatisfactory and limited to slow-fast systems.
method Formulate slow invariant manifolds geometrically within the context of differential geometry, focusing on covariant formulations.
result A more general definition of slow invariant manifolds is provided, independent of coordinate choice.
Paper shows how SFA fits into FBM framework for time series separation.
problem Identifying time series decomposition in flow-based models.
method Combining SFA and FBM to make time series decomposition identifiable.
result Time series decomposition becomes identifiable using SFA and FBM.
Quasiclassical generalized Weierstrass representation for highly corrugated surfaces with slow modulation in the three-dimensional space is proposed. Integrable deformations of such surfaces are described by the dispersionless Veselov-Novikov hierarchy.
The study examines 3-manifolds with slow scalar curvature decay and finds Whitehead manifold properties.
problem Investigating open simply-connected 3-manifolds with slow decay of positive scalar curvature.
method Analyzing topological properties and using Whitehead manifold results.
result Open simply-connected 3-manifolds with the specified properties are homeomorphic to S2imesR. The paper studies invariant complex manifolds in holomorphic slow-fast systems.
problem Existence of invariant complex manifolds in holomorphic systems.
method Geometric singular perturbation theory, Fenichel and Briot-Bouquet theories.
result Conditions are provided to guarantee the existence of one-dimensional invariant complex manifolds.
A single slow-growing tree matches Random Forest's performance.
problem Matching Random Forest's performance with a single tree.
method SGT uses a learning rate to tame CART's greedy algorithm, improving on greedy ML algorithms.
result SGT and tree ensembles like Booging, BT, and RF improve performance.
Researchers reconstruct stiffness tensors from limited data in anisotropic elasticity.
problem Reconstructing stiffness tensors from partial data around one polarization.
method Using algebraic geometry and slowness surfaces, the approach leverages the algebraic geometry of families of slowness surfaces.
result For tensors in a dense open subset, a small amount of data around one polarization uniquely determines the entire slowness surface and stiffness tensor.
This work learns effective dynamics from short-term data of stochastic systems.
problem Learning effective dynamics from short-term data of stochastic systems.
method Proposes a novel algorithm using a neural network (Auto-SDE) to learn invariant slow manifold from data.
result Validated through numerical experiments to be accurate, stable, and effective.
Hierarchical pretraining with slow-fast ODEs
problem Causal self-attention vs. slow-fast ODEs
method Instantiating fast-slow ODE formalism as a concrete neural network
result Equilibrium manifold x=φ(y) is exactly the master-equation (ME) stationary distribution HiGSFA improves SFA by preserving information, enhancing age estimation from facial photos.
problem Discarding useful information prematurely in GSFA networks.
method HiGSFA extends GSFA by incorporating information preservation alongside slowness maximization.
result Achieved a mean absolute error of 3.50 years in estimating human age from facial photographs.
This thesis optimizes neuromorphic systems by slowing down their dynamics, improving performance.
problem Timescale mismatch between analog neuromorphic circuits and real-time sensory inputs.
method Proposes and tests solutions to slow down the dynamics of spiking neural networks.
result Spiking neural networks on analog neuromorphic systems can achieve significant performance boosts.
Efficiently trains forward processes to minimize generative trajectories curvature.
problem High curvature of generative trajectories slows down sampling speed.
method Trains forward process to minimize curvature without ODE/SDE simulation.
result Lower curvature than previous models, decreased sampling costs.
Time-lagged autoencoders improve molecular dynamics data analysis.
problem Analyzing slow collective variables in molecular kinetics.
method Modified autoencoder neural network for dimension reduction.
result Time-lagged autoencoders reliably capture slow dynamics.
Optimizes convex functions in finite vs infinite dimensions, revealing slow convergence rates.
problem Analyzing gradient flows in finite and infinite-dimensional Hilbert spaces.
method Proves convergence rates and optimality conditions for gradient flows and related methods.
result Gradient flow convergence rates in finite dimensions are slower than in infinite dimensions, with optimal rates achievable in Hilbert spaces.
Slow feature analysis (SFA) is a method for extracting slowly varying driving forces from quickly varying nonstationary time series. We show here that it is possible for SFA to detect a component which is even slower than the driving force itself (e.g. the envelope of a modulated sine wave). It is shown that it depends…
Adaptive distributed SGD reduces delay in slow workers.
problem Minimizing delay in distributed SGD with stragglers.
method Adaptive policy for varying k to optimize error-runtime trade-off. result Numerical simulations confirm the effectiveness of the adaptive approach.
New algorithms for faster Schatten quasi-norm minimization.
problem Efficiently approximate matrix rank for large-scale problems.
method Define and solve tractable Schatten quasi-norms, design efficient algorithms.
result Proven algorithms are orders of magnitude faster and more accurate.