Paper discovers structural dynamics equations from only acceleration data.
problem Discovering equations from only acceleration measurements in structural dynamics.
method Library-based approach with Approximate Bayesian Computation (ABC) prioritizing parsimonious models.
result Efficacy demonstrated in four structural dynamics examples, including linear and nonlinear systems.
This paper introduces an acceleration structure for hyperbolic embeddings.
problem Efficiently embedding and visualizing high-dimensional data in hyperbolic spaces.
method Building upon a polar quadtree, the paper introduces a new acceleration structure for hyperbolic embeddings.
result The new method computes embeddings in significantly less time compared to existing methods.
Continuized Nesterov acceleration accelerates stochastic gradient descent and gossip algorithms.
problem Improving the convergence rate of stochastic gradient descent and gossip algorithms.
method Introducing a continuized variant of Nesterov acceleration, which mixes variables continuously and takes gradient steps at random times.
result The continuized Nesterov acceleration achieves convergence rates similar to Nesterov's original acceleration but with random parameters.
New method accelerates gradient descent on curved spaces.
problem Optimizing functions on curved Riemannian manifolds.
method Developed a novel geometric inequality to control metric distortion, enabling a Riemannian accelerated gradient method.
result Proposed the first global accelerated gradient method for Riemannian manifolds.
Accelerates sampling from Gibbs distributions using ARWP method.
problem Sampling from Gibbs distributions efficiently.
method ARWP method, combining Nesterov acceleration and regularized Wasserstein proximal.
result ARWP exhibits higher contraction rate and faster tail exploration.
Accelerates machine learning algorithms for sparse data.
problem Efficiently solving composite convex minimization problems.
method Accelerated dual-averaging primal-dual method for composite convex minimization.
result Demonstrates advantages in handling sparse data both theoretically and empirically.
Motivated by big data applications, first-order methods have been extremely popular in recent years. However, naive gradient methods generally converge slowly. Hence, much efforts have been made to accelerate various first-order methods. This paper proposes two accelerated methods towards solving structured linearly co…
Unified framework for accelerating DNNs on resource-limited platforms.
problem Accelerating DNN execution on resource-limited platforms.
method Block-based pruning framework with reweighted regularization.
result First universal framework for both CNNs and RNNs with real-time acceleration and no accuracy compromise.
LA-VDM accelerates VDM using landmarks to improve data analysis.
problem Efficiently analyzing complex datasets with nonuniform sampling densities.
method Landmark-constrained two-stage normalization to accelerate VDM.
result LA-VDM accurately recovers parallel transport and converges to the connection Laplacian.
In this paper, we propose a novel progressive parameter pruning method for Convolutional Neural Network acceleration, named Structured Probabilistic Pruning (SPP), which effectively prunes weights of convolutional layers in a probabilistic manner. Unlike existing deterministic pruning approaches, where unimportant weig…
Sparsity helps reduce the computational complexity of deep neural networks by skipping zeros. Taking advantage of sparsity is listed as a high priority in next generation DNN accelerators such as TPU. The structure of sparsity, i.e., the granularity of pruning, affects the efficiency of hardware accelerator design as w…
This paper accelerates sparse CNN layers on GPUs by using unstructured sparsity.
problem Efficiency of sparse CNN layers on GPUs.
method Direct sparse operation and reduced precision.
result Achieving up to 90% sparsity in deep CNN models improves efficiency.
In this work, we present an extension of Gaussian process (GP) models with sophisticated parallelization and GPU acceleration. The parallelization scheme arises naturally from the modular computational structure w.r.t. datapoints in the sparse Gaussian process formulation. Additionally, the computational bottleneck is …
SparseRT accelerates sparse computations on GPUs for deep learning inference.
problem Efficiently handling unstructured sparsity patterns on GPUs for deep learning.
method SparseRT, a code generator that leverages unstructured sparsity for accelerating sparse linear algebra operations.
result Geometric mean speedups of 3.4x at 90% sparsity and 5.4x at 95% sparsity for 1x1 convolutions and fully connected layers.
In this work we propose a differential geometric motivation for Nesterov's accelerated gradient method (AGM) for strongly-convex problems. By considering the optimization procedure as occurring on a Riemannian manifold with a natural structure, The AGM method can be seen as the proximal point method applied in this cur…
Arguably, the two most popular accelerated or momentum-based optimization methods in machine learning are Nesterov's accelerated gradient and Polyaks's heavy ball, both corresponding to different discretizations of a particular second order differential equation with friction. Such connections with continuous-time dyna…
This paper analyzes two Lie group momentum optimization algorithms and their convergence rates.
problem Optimizing functions on Lie groups using momentum-based dynamics.
method Investigates Lie Heavy-Ball and Lie NAG-SC algorithms, quantifying their convergence rates under smoothness and convexity assumptions.
result Lie NAG-SC accelerates optimization over the momentumless case, while Lie Heavy-Ball does not.
Accelerates TPP sampling with speculative decoding for faster sequence generation.
problem Efficiently sampling from complex temporal point processes.
method Adapting speculative decoding techniques from language models to TPPs.
result Achieves significant speedup (2-6x) while maintaining distributional accuracy.
In this work we consider the stochastic minimization of nonsmooth convex loss functions, a central problem in machine learning. We propose a novel algorithm called Accelerated Nonsmooth Stochastic Gradient Descent (ANSGD), which exploits the structure of common nonsmooth loss functions to achieve optimal convergence ra…
Nested Slice Sampling accelerates Nested Sampling for GPU acceleration.
problem Challenging inference for complex, multimodal targets.
method Vectorized Nested Slice Sampling using Hit-and-Run Slice Sampling.
result NSS maintains accurate evidence estimates and high-quality posterior samples, robust on multimodal problems.
Surrogate models speed up RL training in dynamic systems.
problem High computational cost of high-fidelity simulations.
method Developed and tested surrogate models for RL training.
result Surrogate models can significantly accelerate RL training.
In recent years, convolutional neural networks (CNNs) have shown great performance in various fields such as image classification, pattern recognition, and multi-media compression. Two of the feature properties, local connectivity and weight sharing, can reduce the number of parameters and increase processing speed dur…
Machine Learning (ML) is making a strong resurgence in tune with the massive generation of unstructured data which in turn requires massive computational resources. Due to the inherently compute- and power-intensive structure of Neural Networks (NNs), hardware accelerators emerge as a promising solution. However, with …
L2O uses ML to optimize traditional optimization techniques.
problem Real-world optimization problems with shared structures.
method Exploiting shared structures to enhance optimization techniques.
result Better or faster solutions through machine learning integration.
This paper is about metric data structures in high-dimensional or non-Euclidean space that permit cached sufficient statistics accelerations of learning algorithms. It has recently been shown that for less than about 10 dimensions, decorating kd-trees with additional "cached sufficient statistics" such as first and sec…
A new k-means algorithm using cover trees accelerates clustering.
problem Efficiently clustering large datasets with k-means.
method Combining cover trees with upper and lower bounds.
result Significantly reduces distance computations and improves clustering performance.
GPU-accelerated particle methods outperform neural samplers in LFT benchmarks.
problem High-dimensional multimodal sampling problems in lattice field theory.
method GPU-accelerated particle Monte Carlo methods (Sequential Monte Carlo and nested sampling).
result These methods match or outperform neural samplers in sample quality and wall-clock time.
State-of-the-art convolutional neural networks (CNNs) used in vision applications have large models with numerous weights. Training these models is very compute- and memory-resource intensive. Much research has been done on pruning or compressing these models to reduce the cost of inference, but little work has address…
LDTA expands LDA's topic modeling capacity with tree-structured priors.
problem Limited expressiveness of Dirichlet priors in LDA for complex topic relationships.
method Introduces Latent Dirichlet-Tree Allocation (LDTA) with Dirichlet-Tree (DT) priors, and develops universal mean-field variational inference and Expectation Propagation.
result LDTA enables expressive, tree-structured priors over topic proportions, expanding modeling capacity of LDA.
Proposes EDM algorithm to accelerate model training in distributed networks.
problem Hindered effectiveness of distributed stochastic optimization algorithms due to data heterogeneity and network sparsity.
method Introduces Exact-Diffusion with Momentum (EDM) algorithm, incorporating momentum techniques to mitigate bias and enhance convergence rate.
result EDM algorithm converges sub-linearly to the optimal solution, radius independent of data heterogeneity, for non-convex objective functions.
Momentum methods such as Polyak's heavy ball (HB) method, Nesterov's accelerated gradient (AG) as well as accelerated projected gradient (APG) method have been commonly used in machine learning practice, but their performance is quite sensitive to noise in the gradients. We study these methods under a first-order stoch…
Accelerates Riemannian gradient methods with extrapolation.
problem Optimizing functions on manifolds efficiently.
method Extrapolating iterates in Riemannian gradient descent.
result Achieves optimal convergence rate and computational advantage.
Sparse coding is a core building block in many data analysis and machine learning pipelines. Typically it is solved by relying on generic optimization techniques, that are optimal in the class of first-order methods for non-smooth, convex functions, such as the Iterative Soft Thresholding Algorithm and its accelerated …
Paper accelerates K-means clustering for large sparse document data.
problem Efficiently clustering large-scale sparse document data.
method Designs an AFM algorithm leveraging UCs and inverted-index structure.
result Significantly improves clustering speed for large-scale documents.
Sparse coding is a core building block in many data analysis and machine learning pipelines. Typically it is solved by relying on generic optimization techniques, such as the Iterative Soft Thresholding Algorithm and its accelerated version (ISTA, FISTA). These methods are optimal in the class of first-order methods fo…
SpinSVAR estimates SVAR models with sparse input, improving accuracy and scalability.
problem Estimating SVAR models with sparse input assumptions.
method SpinSVAR models input as independent Laplacian variables, enforcing sparsity and using least absolute error regression.
result SpinSVAR outperforms state-of-the-art methods in accuracy and runtime, identifying significant structural shocks.
Python toolbox for causal structure learning from data.
problem Causal structure learning from data.
method Generates data from simulators or real-world datasets, learns causal structure, evaluates graphs, and includes gradient-based methods.
result Convenience and efficiency in causal discovery with GPU acceleration.
Accelerates optimization in asynchronous systems with sparse updates.
problem Optimizing finite-sum objectives in asynchronous lock-free environments.
method New accelerated SVRG variant with sparse updates.
result Achieves optimal incremental gradient complexity.
Gait event detection of the initial contact and toe off is essential for running gait analysis, allowing the derivation of parameters such as stance time. Heuristic-based methods exist to estimate these key gait events from tibial accelerometry. However, these methods are tailored to very specific acceleration profiles…
We analyze Riemannian accelerated methods using a new framework.
problem Understanding Riemannian accelerated gradient methods.
method Riemannian A-HPE framework, focusing on Euclidean A-HPE insights and metric distortion control.
result Characterization of acceleration for various Riemannian methods.
Survey of Graph Neural Networks for efficient computation.
problem Efficient processing of Graph Neural Networks (GNNs) is challenging.
method Review of GNN algorithms, software and hardware acceleration analysis.
result Distilled hardware-software, graph-aware, and communication-centric vision for GNN accelerators.
Derives PF-ODE for infinite-dimensional functions, improving function generation tasks.
problem Efficient inference in infinite-dimensional diffusion models.
method Derives PF-ODE in infinite-dimensional function spaces.
result Reduces function evaluations while maintaining sample quality.
Super-acceleration of gradient descent with momentum improves loss function minimization.
problem Minimizing loss functions in machine learning.
method Extending Nesterov acceleration by using gradients at multiple steps ahead.
result Super-acceleration of the momentum algorithm is beneficial for various loss landscapes and tasks.
PF-LaCG removes the need for knowing smoothness and strong convexity parameters for locally accelerated CG.
problem Locally accelerated CG requires knowledge of smoothness and strong convexity parameters.
method Parameter-Free Locally Accelerated CG (PF-LaCG) algorithm.
result PF-LaCG achieves local acceleration without requiring knowledge of smoothness and strong convexity parameters.
Two new algorithms improve neural architecture search efficiency.
problem Optimizing neural architecture search for faster and more accurate models.
method Introduces NASGD and NASAGD using accelerated gradient descent on a semi-discrete space.
result Achieves comparable accuracy with 40x fewer architectures in 12 hours.
Accelerates coordinate descent methods for machine learning problems.
problem Slowness of coordinate descent methods in machine learning.
method Extrapolation-based accelerated coordinate descent.
result Significant speed-up in practice compared to existing methods.
Chebyshev steps improve convergence in deep-unfolded gradient descent.
problem Improving convergence speed in iterative algorithms.
method Introducing Chebyshev steps to bound convergence rate of gradient descent.
result Chebyshev steps lead to asymptotically optimal convergence rate.
Accelerated gradient methods play a central role in optimization, achieving optimal rates in many settings. While many generalizations and extensions of Nesterov's original acceleration method have been proposed, it is not yet clear what is the natural scope of the acceleration concept. In this paper, we study accelera…