The paper improves Gibbs sampling for large graphs by minibatching.
problem High computational cost of single Gibbs sampling update step.
method Minibatching: subsampling factors to estimate their sum.
result Minibatched Gibbs can be made unbiased and converge faster.
Parallelizes LARS for high-dimensional data with speedups and accuracy trade-offs.
problem Fitting linear regression models to high-dimensional data efficiently.
method Two parallel and communication avoiding versions of LARS: bLARS and Tournament-bLARS.
result Speedups up to 4x compared to LARS, with trade-offs in solution quality.
Quantum algorithms speed up derivative pricing beyond Black-Scholes models.
problem Quantum speedups for derivative pricing beyond Black-Scholes models.
method Utilizing fast-forwardability and quantum Milstein sampler for non-GBM models, and improved numerical integration for GBM and CIR models.
result Quadratic speedups for derivative pricing in practical models like CIR and Heston's model.
A new algorithm speeds up rerandomization for better experiment balance.
problem Achieving optimal covariate balance in randomized experiments.
method Metropolis-Hastings framework with sampling-importance resampling.
result PSRSRR achieves significant speedups while maintaining statistical guarantees.
Improved neural framework for scaling entropic MOT with significant computational gains.
problem High computational overhead in multimarginal optimal transport.
method Neural Entropic MOT (NEMOT) using mini-batch training to reduce complexity.
result Significant speedups and feasibility improvements for multimarginal data.
ProxSkip achieves linear speedup in distributed non-convex optimization.
problem Achieving linear speedup in distributed non-convex optimization.
method Unified convergence analysis for stochastic non-convex, convex, and strongly convex problems.
result ProxSkip achieves linear speedup in the number of nodes under stochastic gradients.
Linear speedup achieved in non-convex optimization for decentralized systems.
problem Achieving optimal performance in decentralized non-convex optimization.
method Examined the dependence of convergence guarantees on spectral properties of combination policies.
result Linear speedup in saddle-point escape time for symmetric combination policies.
Rk-means clusters relational data without full matrix, speeding up clustering.
problem Clustering relational data without full matrix computation.
method Constructs a grid coreset for clustering, avoiding expensive feature extraction queries.
result Orders-of-magnitude speedup in clustering relational data.
Paper speeds up structured output prediction without sacrificing accuracy.
problem Computational inefficiency in predicting structured outputs.
method Learning to search approach to train a speedup classifier.
result Speedup classifier outperforms greedy search in terms of speed.
Quantum computing promises faster finance algorithms.
problem Solving finance problems faster than classical methods.
method Quantum computing applications to finance, including Monte Carlo, portfolio optimization, and machine learning.
result Quantum speedups for finance problems, especially Monte Carlo and portfolio optimization.
A fast method for Lasso and Logistic Lasso problems.
problem Solving Lasso and Logistic Lasso regression problems efficiently.
method Iterative active set approach using solver updates.
result 31.41 times faster on average for compressed sensing.
New insights explain speedup saturation in distributed learning with large batches and delays.
problem Understanding and optimizing speedup in distributed learning with large batches and delays.
method Theoretical analysis of strongly convex, convex, and non-convex settings, considering data sparsity.
result Identification of a data-dependent parameter explaining speedup saturation in both batch size and gradient staleness.
Quantum computing offers a quadratic speedup for estimating non-linear functionals.
problem Estimating non-linear functionals of probability distributions.
method Proposes a quantum-inside-quantum Monte Carlo algorithm for a broad class of non-linear estimation problems.
result Achieves a quadratic speedup for non-linear estimation problems, including nested conditional expectations and stochastic optimization.
Quantum algorithm speeds up nested expectation estimation by nearly quadratically.
problem Estimating repeatedly nested expectations with quantum computing.
method Proposes a quantum algorithm achieving nearly quadratic speedup over classical methods.
result Achieves nearly quadratic speedup for RNEs, up to logarithmic factors.
Federated Q-Learning achieves linear regret speedup with low communication cost.
problem Achieving linear regret speedup in federated reinforcement learning without high communication costs.
method Proposed two federated Q-Learning algorithms: FedQ-Hoeffding and FedQ-Bernstein, using event-triggered synchronization, novel step size selection, and concentration inequalities.
result Total regrets achieve linear speedup compared to single-agent counterparts with logarithmic communication cost.
A new algorithm speeds up deep learning training by decoupling computation and communication.
problem High communication cost limits the speedup of distributed SGD.
method CoCoD-SGD: runs computation and communication in parallel.
result Linear time speedup with respect to hardware resources.
GPU optimization speeds up large-scale classification tasks.
problem Efficiently training large-scale classification models on GPUs.
method Judecious GPU-optimization principles applied to TRON algorithm.
result Significant speedups for logistic regression and SVM classification.
We study nonconvex finite-sum problems and analyze stochastic variance reduced gradient (SVRG) methods for them. SVRG and related methods have recently surged into prominence for convex optimization given their edge over stochastic gradient descent (SGD); but their theoretical analysis almost exclusively assumes convex…
Optimized parallel RNN training reaches up to 845x speedup.
problem Expensive RNN training through back-propagation through time (BPTT).
method Optimized parallel algorithm \opt based on ELM, leveraging GPU shared memory and QR factorization.
result Up to 845x speedup over sequential training and 20x less time to train.
Neural network speeds up atmospheric chemistry modeling 4250x.
problem Computational expense of simulating atmospheric chemistry.
method Created a neural network to emulate a complex chemical mechanism.
result Achieved a 250x computational speedup.
New algorithm achieves linear speedup in non-i.i.d. federated bilevel learning.
problem Linear speedup in convergence for non-i.i.d. datasets in federated bilevel optimization.
method Proposes FedMBO with a novel client sampling scheme for non-i.i.d. datasets.
result Achieves a convergence rate of O(1/√(nK) + 1/K + √n/K³/²).
System learns optimizer hyperparameters to generalize across tasks.
problem Lack of generalization in learning optimizers for neural networks.
method Generalization-first approach, learning optimizer hyperparameters.
result System outperforms Adam on various tasks, including unseen modalities.
Improved reinforcement learning with emergency stops.
problem Reducing exploration in reinforcement learning.
method Emergency stop mechanisms to reduce sample complexity.
result Significant improvement in sample complexity and speed.
ECD algorithm speeds up non-convex optimization, offering quantum and stochastic enhancements.
problem Non-convex optimization challenges in machine learning.
method Energy Conserving Descent (ECD) algorithm, stochastic ECD dynamics (sECD), quantum ECD Hamiltonian (qECD).
result ECD and its quantum version achieve exponential speedup over gradient descent.
Unified analysis of Federated Averaging and Nesterov FedAvg for linear speedup.
problem Understanding convergence of FL algorithms under non-i.i.d. data and partial participation.
method Systematic study of convergence guarantees for FedAvg and Nesterov FedAvg under different conditions.
result Unified analysis of linear speedup for FedAvg and Nesterov FedAvg in various settings.
CHEETAH speeds up secure MLaaS by 100x over fastest existing schemes.
problem Privacy-preserving machine learning on end devices.
method Ultra-fast secure MLaaS framework using secret sharing.
result More than 100x speedup over fastest existing schemes.
Investigates the impact of batch size on GPU and TPU performance.
problem Optimizing performance of GPUs and TPUs during training and inference phases.
method Investigated the impact of batch size on performance of GPUs and TPUs using standard MNIST and Fashion-MNIST datasets.
result Significant speedup was achieved even with low-scale usage of TPUv2 units, up to 10x for training and 2x for prediction.
A central task in the field of quantum computing is to find applications where quantum computer could provide exponential speedup over any classical computer. Machine learning represents an important field with broad applications where quantum computer may offer significant speedup. Several quantum algorithms for discr…
ML-EM method speeds up diffusion model sampling.
problem Efficiently sampling from complex diffusion models.
method Multilevel Euler-Maruyama method with UNet approximations.
result Polynomial speedup in sampling from diffusion models.
We develop parallel and distributed Frank-Wolfe algorithms; the former on shared memory machines with mini-batching, and the latter in a delayed update framework. Whenever possible, we perform computations asynchronously, which helps attain speedups on multicore machines as well as in distributed environments. Moreover…
Paper analyzes convergence and speedup of asynchronous parallel SGD.
problem Achieving good convergence and linear speedup in asynchronous parallel SGD.
method Second-order convergence analysis of APSGD with consistent read near strictly saddle points.
result Theoretical guarantee for using at most O ( K 1 / 3 M − 1 / 3 ) O(K^{1/3}M^{-1/3}) O ( K 1/3 M − 1/3 ) workers for good convergence and linear speedup. Novel periodic momentum SGD method for decentralized training with linear speedup.
problem Lack of effective momentum schema in decentralized training methods.
method Proposes a novel periodic decentralized momentum SGD method.
result Achieves linear speedup in decentralized training.
Speed up neural networks by 2x with 5% mAP loss.
problem High computational cost of neural network forward passes.
method Deep Learning Approximation: lossless and lossy optimizations.
result 2x speedup in network forward pass with 5% mAP drop.
We present the Parallel, Forward-Backward with Pruning (PFBP) algorithm for feature selection (FS) in Big Data settings (high dimensionality and/or sample size). To tackle the challenges of Big Data FS PFBP partitions the data matrix both in terms of rows (samples, training examples) as well as columns (features). By e…
Quantum algorithm speeds up MIP solving by a near-quadratic factor.
problem Solving Mixed Integer Programs (MIPs) efficiently.
method Incremental-Quantum-Branch-and-Bound algorithm combining quantum speedup with classical search heuristics.
result Universal near-quadratic speedup over classical Branch-and-Bound algorithms.
Quantum algorithm speeds up learning from big data exponentially.
problem Scalable learning from big data with optimized random features.
method Quantum algorithm for sampling optimized random features.
result Exponential speedup in runtime compared to classical algorithms.
Speeds up deep neural networks training by 10x using GPU concurrency.
problem Training deep residual neural networks efficiently.
method Layer-wise parallel training with GPU concurrency and Nonlinear Multigrid.
result 10.2x speedup over traditional techniques.
Federated learning algorithm improves with intermittent client availability.
problem Performance degradation in Federated Averaging due to client availability changes.
method Federated Latest Averaging (FedLaAvg) uses latest gradients from all clients, even when unavailable.
result FedLaAvg achieves sublinear speedup compared to classical Federated Averaging.
Paper studies collaboration in multi-armed bandits with limited interaction.
problem Identifying the best arm collaboratively with limited communication.
method Developed techniques to quantify and prove round-speedup tradeoffs.
result Almost tight round-speedup tradeoffs for distributed exploration.
WU-UCT parallelizes MCTS with linear speedup and limited performance loss.
problem Challenges in parallelizing Monte Carlo Tree Search (MCTS) due to its sequential nature.
method Introduces unobserved samples to track incomplete simulations and modify UCT tree policy.
result Achieves linear speedup and only limited performance loss with increasing parallel workers.
New AMSGrad variant accelerates nonconvex optimization.
problem Training deep neural networks efficiently.
method Adds prior knowledge to AMSGrad, leveraging sequential predictability and optimistic online learning.
result Accelerates convergence and increases sample efficiency.
Asynchronous L-BFGS speeds up non-convex optimization.
problem Non-convex optimization challenges in machine learning.
method Asynchronous stochastic L-BFGS algorithm for non-convex optimization.
result Achieves an ergodic convergence rate of O ( 1 / N ) {\cal O}(1/\sqrt{N}) O ( 1/ N ) and linear speedup. We parallelize backpropagation for deep learning models, achieving significant speedups.
problem Sequential dependency in backpropagation limits scalability on parallel systems.
method Reformulated backpropagation as a scan operation, using Blelloch scan algorithm.
result Up to 2.75x speedup on overall training time and 108x on backward pass.
A faster method for estimating effects in large data using fixed-point trees.
problem Estimating heterogeneous effects in large dimensions with computational efficiency.
method Fixed-point approximation to eliminate Jacobian estimation and speed up GRFs.
result Significant computational efficiency improvement without sacrificing statistical accuracy.
New GPU kernels boost deep learning speed and memory efficiency.
problem Sparse deep learning matrices are not well-suited for existing sparse kernels.
method Identified favorable properties of sparse matrices from deep learning, developed high-performance GPU kernels for sparse matrix operations.
result 27% of single-precision peak performance on Nvidia V100 GPUs achieved with new kernels.
UD-SGD analysis shows efficient sampling by a few agents can outperform others.
problem Analyzing convergence speed and sampling strategies in UD-SGD.
method Asymptotic analysis of UD-SGD with various communication patterns and sampling strategies.
result Efficient sampling by a few agents can lead to better overall convergence.
Efficiently approximates Sparse PCA with significant speedups and minor error.
problem Sparse Principal Component Analysis (Sparse PCA) is NP-hard and computationally expensive.
method Approximates the covariance matrix with block-diagonal form, solves sub-problems in each block, and reconstructs the solution.
result Significant computational speedups with minor additive error.
SliceOut speeds up deep learning training without sacrificing accuracy.
problem Frequent model re-training and large model training workloads in deep learning.
method SliceOut uses dropout-inspired scheme to drop contiguous sets of units at random, leveraging GPU memory layout.
result 10-40% speedups and memory reduction with minimal accuracy loss.