ErasureHead speeds up distributed GD with approximate gradient coding.
problem Mitigating delays in distributed gradient descent.
method Approximate gradient coding to tolerate delays.
result ErasureHead converges as quickly as GD and has faster runtime under probabilistic delays.
Gradient coding is a technique for straggler mitigation in distributed learning. In this paper we design novel gradient codes using tools from classical coding theory, namely, cyclic MDS codes, which compare favorably with existing solutions, both in the applicable range of parameters and in the complexity of the invol…
Gradient codes use block designs to resist adversarial stragglers in distributed computing.
problem Mitigating slow machines (stragglers) in distributed gradient-based methods.
method Gradient coding based on balanced incomplete block designs (BIBDs) to resist adversarial selection of stragglers.
result Adversarial stragglers have no advantage over random selection, and codes based on symmetric BIBDs maximize the adversarial threshold.
New gradient coding schemes reduce decoding error in both random and adversarial straggler settings.
problem Creating efficient approximate gradient coding schemes for distributed optimization.
method Introduced novel approximate gradient codes based on expander graphs, achieving optimal decoding coefficients.
result Achieved nearly optimal error in random setting and nearly half the error in adversarial setting compared to existing codes.
Distributed algorithms are often beset by the straggler effect, where the slowest compute nodes in the system dictate the overall running time. Coding-theoretic techniques have been recently proposed to mitigate stragglers via algorithmic redundancy. Prior work in coded computation and gradient coding has mainly focuse…
This work improves federated learning privacy and accuracy with non-private data sharing and approximate gradient coding.
problem Challenges of non-IID data and stragglers in federated learning.
method Data-driven strategy combining offline data sharing and approximate gradient coding.
result Achieves a trade-off between privacy and utility, leading to improved model convergence and accuracy.
Paper proposes new gradient codes for robust distributed machine learning.
problem Robustness against slow machines (stragglers) in distributed machine learning.
method Constructs gradient codes based on probabilistic and Kronecker product methods.
result Gradient codes exist for a wide range of system parameters with comparable error performance.
This work improves dictionary learning speed without sacrificing accuracy.
problem Prohibitive computational cost of standard dictionary learning methods.
method Approximate dictionary learning using unrolling and gradient descent.
result Unrolling outperforms standard methods in support estimation and early iterations.
Gradient codes adapt to varying straggler counts in distributed learning.
problem Mitigating slow worker nodes (stragglers) in distributed machine learning.
method Proposes a flexible gradient coding scheme that concatenates codes for different straggler tolerances, adapting to actual straggler counts.
result Significantly lower latency compared to fixed-tolerance gradient codes.
Hybrid approach reduces computation time and decoding complexity.
problem Straggling servers in distributed computing.
method Coded partial gradient computation (CPGC) that balances gradient accuracy and completion time.
result Reduces both computation time and decoding complexity.
STRATA generates code adversarial examples efficiently without gradients.
problem Generating adversarial examples for code that retains functional meaning.
method Uses token frequency statistics to construct gradient-free adversarial examples.
result Empirically outperforms gradient-based methods with less information and effort.
BayesDLL offers a PyTorch library for Bayesian deep learning with large models.
problem Bayesian inference for large-scale deep networks.
method Variational inference, MC-dropout, stochastic-gradient MCMC, Laplace approximation.
result BayesDLL can handle Vision Transformers and pre-trained model weights as priors.
Gradient descent and its many variants, including mini-batch stochastic gradient descent, form the algorithmic foundation of modern large-scale machine learning. Due to the size and scale of modern data, gradient computations are often distributed across multiple compute nodes. Unfortunately, such distributed implement…
Modern large scale machine learning applications require stochastic optimization algorithms to be implemented on distributed computational architectures. A key bottleneck is the communication overhead for exchanging information such as stochastic gradients among different workers. In this paper, to reduce the communica…
BackPACK extends PyTorch to compute additional gradient info.
problem Lack of efficient tools for computing mini-batch variance and Hessian approximations.
method BackPACK builds on PyTorch to automatically compute additional derivatives.
result BackPACK enables efficient computation of various derivative quantities.
Proposes a new CSC model for handling unknown noise.
problem Existing CSC methods can only model Gaussian noise, which is restrictive.
method Uses Gaussian mixture model for unknown noise and EM algorithm for optimization.
result Effective modeling of complicated unknown noise with high-quality filters and representation.
This paper explores how random sampling and coding can speed up approximate matrix multiplication.
problem Efficiently computing large-scale matrix multiplications in distributed systems.
method Proposes two schemes: coding for recovery and random sampling for approximation.
result Investigates tradeoffs between recovery threshold and approximation error.
We propose a novel coding theoretic framework for mitigating stragglers in distributed learning. We show how carefully replicating data blocks and coding across gradients can provide tolerance to failures and stragglers for Synchronous Gradient Descent. We implement our schemes in python (using MPI) to run on Amazon EC…
New method for scalable set encoding with unbiased gradient approximation.
problem Limited expressive power and large set training issues in set functions.
method Universally MBC (UMBC) class of set functions and efficient MBC training algorithm.
result Unbiased approximation of full set gradient with constant memory overhead.
New loss function and training scheme improve binary hash codes for better similarity search.
problem Improving binary hash codes for better similarity search tasks.
method Log likelihood loss on Hamming distance target, novel training scheme, multi-indexing.
result Significant improvements in MAP (84%) and query cost reduction for ImageNet and SIFT 1M.
This paper develops coding techniques to reduce the running time of distributed learning tasks. It characterizes the fundamental tradeoff to compute gradients (and more generally vector summations) in terms of three parameters: computation load, straggler tolerance and communication cost. It further gives an explicit c…
Paper proposes a method to numerically approximate differential privacy guarantees using FFT.
problem Computing tight differential privacy guarantees for machine learning algorithms.
method The method uses numerical approximation of an integral formula, discretization, and the fast Fourier transform algorithm.
result Significant improvements in bound tightness and computation time compared to state-of-the-art techniques.
SGP combines PushSum with stochastic gradient updates for robust distributed deep learning.
problem Synchronization issues in distributed deep learning.
method Stochastic Gradient Push (SGP) using PushSum for approximate distributed averaging.
result SGP converges to a stationary point at the same rate as SGD and achieves consensus.
PPC learns binary codes from data similarities and dissimilarities.
problem Creating efficient binary codes from data similarities and dissimilarities.
method PPC learns binary codes by modeling attractive and repulsive forces in a signed graph.
result PPC achieves superior results in nearest-neighbor searches compared to spectral methods.
Slow running or straggler tasks can significantly reduce computation speed in distributed computation. Recently, coding-theory-inspired approaches have been applied to mitigate the effect of straggling, through embedding redundancy in certain linear computational steps of the optimization algorithm, thus completing the…
In order to interact intelligently with objects in the world, animals must first transform neural population responses into estimates of the dynamic, unknown stimuli which caused them. The Bayesian solution to this problem is known as a Bayes filter, which applies Bayes' rule to combine population responses with the pr…
We use matricial free energy to regularize autoencoders, producing Gaussian-like codes.
problem Generating Gaussian-like codes for autoencoders.
method Define a differentiable loss function based on singular values of the code matrix, minimizing matricial free energy.
result Minimizing matricial free energy results in Gaussian-like codes that generalize.
This work tackles sparse coding in DLRA for interpretable multiway data.
problem Sparse coding in DLRA for interpretable multiway data.
method Proposes a new sparse-coding subproblem (MSC) and several algorithms to solve it.
result DLRA extends low-rank approximations, reducing variance and enhancing interpretability.
A new method directly encodes data into latent space using gradient flow.
problem Suboptimal representations in physical sciences due to encoder inversion.
method Decoder-only approach using gradient flow and ODEs, avoiding integrals.
result Superior data efficiency and explicit encoding compared to traditional autoencoders.
This paper considers the problem of implementing large-scale gradient descent algorithms in a distributed computing setting in the presence of {\em straggling} processors. To mitigate the effect of the stragglers, it has been previously proposed to encode the data with an erasure-correcting code and decode at the maste…
Inspired by recent work on convex formulations of clustering (Lashkari & Golland, 2008; Nowozin & Bakir, 2008) we investigate a new formulation of the Sparse Coding Problem (Olshausen & Field, 1997). In sparse coding we attempt to simultaneously represent a sequence of data-vectors sparsely (i.e. sparse approximation (…
A new method for efficient computation of Knowledge Gradient in Bayesian optimization.
problem Efficient computation of the Knowledge Gradient for Bayesian optimization.
method One-shot Hybrid KG, a new approach combining previous ideas.
result The new method is cheap to compute and preserves theoretical properties of previous methods.
Automatic differentiation (AD) is an essential primitive for machine learning programming systems. Tangent is a new library that performs AD using source code transformation (SCT) in Python. It takes numeric functions written in a syntactic subset of Python and NumPy as input, and generates new Python functions which c…
New entropy-based objective for sparse coding improves learning.
problem Sparse coding with probabilistic priors and non-Gaussian observables.
method Derive a solely entropy-based learning objective for sparse coding parameters.
result Fully analytical ELBO objective for sparse coding with non-trivial posterior approximations.
CodedReduce combines tree topology and gradient coding for efficient and resilient gradient aggregation.
problem Efficient and robust gradient aggregation in distributed learning.
method CodedReduce combines tree topology and gradient coding to overcome bandwidth bottlenecks and straggler delays.
result CodedReduce achieves up to 27.2x speedup over benchmarks GC and RAR.
Study on reducing forgetting in neural networks using compression theory.
problem Catastrophic forgetting in neural networks.
method Defined forgetting as increased description lengths, compared variational posterior approaches to prequential coding methods.
result Proposed a new continual learning method combining ML plug-in and Bayesian mixture codes.
Locally Optimal Block Preconditioned Conjugate Gradient (LOBPCG) is demonstrated to efficiently solve eigenvalue problems for graph Laplacians that appear in spectral clustering. For static graph partitioning, 10-20 iterations of LOBPCG without preconditioning result in ~10x error reduction, enough to achieve 100% corr…
Improves autoencoder reconstruction quality by approximating latent space with adversarial learning.
problem Ambiguity in autoencoder reconstructions and difficulty in matching true distribution.
method Adversarially Approximated Autoencoder (AAAE) using GAN for flexible latent space approximation.
result Generates more faithful reconstructions and maintains latent manifold structure.
DiffTaichi enables fast, differentiable physical simulations with shorter code.
problem Building efficient differentiable physical simulators.
method Differentiable programming language (DiffTaichi) that generates gradients using source code transformations and a light-weight tape.
result Differentiable physical simulators written in DiffTaichi are faster and more concise than existing methods.
A new algorithm reduces the computational cost of RTRL while maintaining performance.
problem Large computational costs in RTRL for learning long-term dependencies.
method Optimal Kronecker-Sum Approximation (OK) of RTRL.
result OK is optimal for a class of RTRL approximations and matches TBPTT in performance.
We present a comprehensive framework for structured sparse coding and modeling extending the recent ideas of using learnable fast regressors to approximate exact sparse codes. For this purpose, we develop a novel block-coordinate proximal splitting method for the iterative solution of hierarchical sparse coding problem…
Sparse codes improve optimal control tasks with correlated inputs.
problem Optimal control tasks with correlated feature inputs.
method Used a sparse code to represent natural images in an optimal control task solved with neuro-dynamic programming.
result An over-complete sparse code increases memory capacity and learning speed beyond a complete code.
A faster method for optimizing DNA and protein sequences using machine learning.
problem Designing DNA and protein sequences with improved function.
method Activation maximization with a straight-through approximation and adaptive entropy variable.
result Fast SeqProp achieves up to 100-fold faster convergence and improved fitness optima.
Sparse coding, which represents a data point as a sparse reconstruction code with regard to a dictionary, has been a popular data representation method. Meanwhile, in database retrieval problems, learning the ranking scores from data points plays an important role. Up to now, these two problems have always been conside…
The paper proposes a method to train time-varying generative models using natural gradients.
problem Training time-varying generative models efficiently and accurately.
method Projecting generative model parameters onto an exponential family manifold and optimizing using natural gradient descent.
result The proposed method efficiently approximates the natural gradient and can be applied to various exponential family models.
D2SRM solves complex PDEs using deep learning.
problem High-dimensional, Hessian-dependent fully nonlinear parabolic PDEs.
method Single scalar space-time network generating derivative-consistent approximations trained through residuals and penalties.
result Well-posedness and convergence theory established for globally Lipschitz equations.
Proposes a non-parametric method for deep discrete latent variable models.
problem Learning sparse discrete latent representations in deep models.
method Iterative algorithm with Beta-Bernoulli process prior and local data scaling.
result Improves sparsity and scalability of deep discrete latent variable models.
ICQ improves high-dimensional similarity search without sacrificing precision.
problem High-dimensional similarity search is computationally expensive.
method Interleaved Composite Quantization (ICQ) reduces code length and quantization error.
result ICQ achieves fast similarity search without using shorter codes.