arXiv research

Ouroboros accelerates training of large Transformer models.

Linear algebra approach for parallel deep learning models.

When training large machine learning models with many variables or parameters, a single machine is often inadequate since the model may be too large to fit in memory, while training can take a long time even with stochastic updates. A natural recourse is to turn to distributed cluster computing, in order to harness add…

Large models are prevalent in modern machine learning scenarios, including deep learning, recommender systems, etc., which can have millions or even billions of parameters. Parallel algorithms have become an essential solution technique to many large-scale machine learning jobs. In this paper, we propose a model parall…

In real world industrial applications of topic modeling, the ability to capture gigantic conceptual space by learning an ultra-high dimensional topical representation, i.e., the so-called "big model", is becoming the next desideratum after enthusiasms on "big data", especially for fine-grained downstream tasks such as …

Mesh-TensorFlow enables efficient deep learning on large clusters.

ZeRO optimizes memory for training large models, scaling to trillions of parameters.

Training deep networks is expensive and time-consuming with the training period increasing with data size and growth in model parameters. In this paper, we provide a framework for distributed training of deep networks over a cluster of CPUs in Apache Spark. The framework implements both Data Parallelism and Model Paral…

This work optimizes deep learning training by combining data and model parallelism.

New types of machine learning hardware in development and entering the market hold the promise of revolutionizing deep learning in a manner as profound as GPUs. However, existing software frameworks and training algorithms for deep learning have yet to evolve to fully leverage the capability of the new wave of silicon.…

TSSM splits neural networks for parallel training with minimal accuracy loss.

Paper proposes DSP to accelerate deep learning model training by addressing locking and straggler problems.

Adaptive batch size schedules improve language model training efficiency and generalization.

Deep neural networks, and in particular recurrent networks, are promising candidates to control autonomous agents that interact in real-time with the physical world. However, this requires a seamless integration of temporal features into the network's architecture. For the training of and inference with recurrent neura…

Scaling multinomial logistic regression to datasets with very large number of data points and classes is challenging. This is primarily because one needs to compute the log-partition function on every data point. This makes distributing the computation hard. In this paper, we present a distributed stochastic gradient d…

Cyclic Data Parallelism reduces memory usage and balances gradient communications.

The paper optimizes dynamic scheduling for ring architectures in deep learning training.

This paper optimizes how deep learning models are distributed across different devices.

Speeds up deep neural networks training by 10x using GPU concurrency.

The training of Deep Neural Networks usually needs tremendous computing resources. Therefore many deep models are trained in large cluster instead of single machine or GPU. Though major researchs at present try to run whole model on all machines by using asynchronous asynchronous stochastic gradient descent (ASGD), we …

Paper proposes DCT for efficient hybrid parallel training of large recommendation models.

Training examples are not all equally informative. Active learning strategies leverage this observation in order to massively reduce the number of examples that need to be labeled. We leverage the same observation to build a generic strategy for parallelizing learning algorithms. This strategy is effective because the …

IST trains neural networks locally, reducing memory and communication costs.

Quadratic models explain neural network behavior during training.

The paper parallelizes HMM inference for efficient long-term computations.

A simple approach speeds up probabilistic programming models.

This work proposes an efficient autoregressive model for text generation.

Two parallel samplers enhance image quality in limited denoising steps.

TF-Replicator simplifies distributed ML for researchers.

New algorithm speeds up spike sorting for large datasets.

PETRA enables parallel training of deep models with reversible architectures.

FasTR efficiently solves sparse and unit-rank tensor regression problems.

Optimizes parallel training of linear models, improving convergence.

There is significant recent interest to parallelize deep learning algorithms in order to handle the enormous growth in data and model sizes. While most advances focus on model parallelization and engaging multiple computing agents via using a central parameter server, aspect of data parallelization along with decentral…

Stochastic variational inference (SVI), the state-of-the-art algorithm for scaling variational inference to large-datasets, is inherently serial. Moreover, it requires the parameters to fit in the memory of a single processor; this is problematic when the number of parameters is in billions. In this paper, we propose e…

New method solves blind inverse problems by optimizing both operator and image parameters.

ShadowSync separates background synchronization for scalable distributed training.

What is a systematic way to efficiently apply a wide spectrum of advanced ML programs to industrial scale problems, using Big Models (up to 100s of billions of parameters) on Big Data (up to terabytes or petabytes)? Modern parallelization strategies employ fine-grained operations and scheduling beyond the classic bulk-…

Proposes ContSup to boost local learning by supplying context between isolated modules.

Performance of distributed optimization and learning systems is bottlenecked by "straggler" nodes and slow communication links, which significantly delay computation. We propose a distributed optimization framework where the dataset is "encoded" to have an over-complete representation with built-in redundancy, and the …

Improves Bayesian neural learning efficiency with surrogate-assisted parallel tempering.

New algorithms reduce bias in federated learning for non-i.i.d. data.

A new parallel clustering method improves speed and accuracy for single cell transcriptomic data.

Recent technological development has enabled researchers to study social phenomena scientifically in detail and financial markets has particularly attracted physicists since the Brownian motion has played the key role as in physics. In our previous report (arXiv:1703.06739; to appear in Phys. Rev. Lett.), we have prese…

NEST optimizes deep learning training by placing devices efficiently across networks and memory.

We parallelize backpropagation for deep learning models, achieving significant speedups.

Predictability enables efficient parallelization of nonlinear models.

Approaches for big data regression using sufficient statistics.