A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
We analyze the Hessian spectra of large models up to 100B parameters.
This paper optimizes deep learning training by efficiently sharding weight updates across replicas.
GShard enables scaling of large neural networks with automatic sharding and lightweight APIs.
SBT model uses randomized sharding and sub-models to improve Bayesian Additive Regression Trees.
A new framework for clustering high-dimensional data using vertical shards.
Adaptive batch size schedules improve language model training efficiency and generalization.
New method ensures consistent inference across different tensor parallel sizes for large language models.
This work proposes a method to optimize hyperparameters without validation data.
NEST optimizes deep learning training by placing devices efficiently across networks and memory.
Paper presents efficient algorithms for convolutional neural networks using Winograd minimal filtering.
GraphEraser improves unlearning efficiency for graph data.
We introduce a new embarrassingly parallel parameter learning algorithm for Markov random fields with untied parameters which is efficient for a large class of practical models. Our algorithm parallelizes naturally over cliques and, for graphs of bounded degree, its complexity is linear in the number of cliques. Unlike…
In federated distributed learning, the goal is to optimize a global training objective defined over distributed devices, where the data shard at each device is sampled from a possibly different distribution (a.k.a., heterogeneous or non i.i.d. data samples). In this paper, we generalize the local stochastic and full gr…
ParaMonte::Python streamlines Bayesian data analysis with fast Monte Carlo and MCMC routines.
End-to-end Text-to-speech (TTS) system can greatly improve the quality of synthesised speech. But it usually suffers form high time latency due to its auto-regressive structure. And the synthesised speech may also suffer from some error modes, e.g. repeated words, mispronunciations, and skipped words. In this paper, we…
In this work we introduce a semi-supervised approach to the voice conversion problem, in which speech from a source speaker is converted into speech of a target speaker. The proposed method makes use of both parallel and non-parallel utterances from the source and target simultaneously during training. This approach ca…
This work proposes an efficient autoregressive model for text generation.
It is time-consuming and error-prone to implement inference procedures for each new probabilistic model. Probabilistic programming addresses this problem by allowing a user to specify the model and having a compiler automatically generate an inference procedure for it. For this approach to be practical, it is important…
Recurrent neural networks (RNNs) are omnipresent in sequence modeling tasks. Practical models usually consist of several layers of hundreds or thousands of neurons which are fully connected. This places a heavy computational and memory burden on hardware, restricting adoption in practical low-cost and low-power devices…
Deep neural networks, and in particular recurrent networks, are promising candidates to control autonomous agents that interact in real-time with the physical world. However, this requires a seamless integration of temporal features into the network's architecture. For the training of and inference with recurrent neura…
New method reconstructs hidden structures from noisy data.
The numerical solution of large-scale PDEs, such as those occurring in data-driven applications, unavoidably require powerful parallel computers and tailored parallel algorithms to make the best possible use of them. In fact, considerations about the parallelization and scalability of realistic problems are often criti…
We introduce a fast model based deep learning approach for calibrationless parallel MRI reconstruction. The proposed scheme is a non-linear generalization of structured low rank (SLR) methods that self learn linear annihilation filters from the same subject. It pre-learns non-linear annihilation relations in the Fourie…
Bayesian optimization has become a popular method for high-throughput computing, like the design of computer experiments or hyperparameter tuning of expensive models, where sample efficiency is mandatory. In these applications, distributed and scalable architectures are a necessity. However, Bayesian optimization is mo…
We describe a computationally efficient, stochastic graph-regularization technique that can be utilized for the semi-supervised training of deep neural networks in a parallel or distributed setting. We utilize a technique, first described in [13] for the construction of mini-batches for stochastic gradient descent (SGD…
A single policy suffices for near-optimal parallel exploration in RL.
Training deep networks is expensive and time-consuming with the training period increasing with data size and growth in model parameters. In this paper, we provide a framework for distributed training of deep networks over a cluster of CPUs in Apache Spark. The framework implements both Data Parallelism and Model Paral…
New algorithms reduce communication in GNN training.
Machine learning methods are widely used for a variety of prediction problems. \emph{Prediction as a service} is a paradigm in which service providers with technological expertise and computational resources may perform predictions for clients. However, data privacy severely restricts the applicability of such services…
TensorOpt finds optimal parallelization strategies for DNN training.
New Feedback Transformer architecture improves model performance by exposing past representations to future.
The study characterizes compact homogeneous manifolds with Bismut parallel torsion.
In this paper, we propose a novel multi-task learning method based on the deep convolutional network. The proposed deep network has four convolutional layers, three max-pooling layers, and two parallel fully connected layers. To adjust the deep network to multi-task learning problem, we propose to learn a low-rank deep…
We propose weight-sharing regularization for neural networks.
Path regularization reveals convex optimization in deep ReLU networks.
Latent Dirichlet Allocation (LDA) is a topic model widely used in natural language processing and machine learning. Most approaches to training the model rely on iterative algorithms, which makes it difficult to run LDA on big corpora that are best analyzed in parallel and distributed computational environments. Indeed…
Scalable3-BO tackles scalability issues in Bayesian optimization for big data and high dimensions.
We study recursive-cube-of-rings (RCR), a class of scalable graphs that can potentially provide rich inter-connection network topology for the emerging distributed and parallel computing infrastructure. Through rigorous proof and validating examples, we have corrected previous misunderstandings on the topological prope…
Training a neural network using backpropagation algorithm requires passing error gradients sequentially through the network. The backward locking prevents us from updating network layers in parallel and fully leveraging the computing resources. Recently, there are several works trying to decouple and parallelize the ba…
Backpropagation algorithm is indispensable for the training of feedforward neural networks. It requires propagating error gradients sequentially from the output layer all the way back to the input layer. The backward locking in backpropagation algorithm constrains us from updating network layers in parallel and fully l…
SAMPa speeds up SAM by parallelizing its computations.
Parallel unlearning framework for inherited models reduces computational overhead.
Single global merging boosts decentralized learning performance.
Space-filling designs such as scrambled-Hammersley, Latin Hypercube Sampling and Jittered Sampling have been proposed for fully parallel hyperparameter search, and were shown to be more effective than random or grid search. In this paper, we show that these designs only improve over random search by a constant factor. …
We propose a new model for unsupervised document embedding. Leading existing approaches either require complex inference or use recurrent neural networks (RNN) that are difficult to parallelize. We take a different route and develop a convolutional neural network (CNN) embedding model. Our CNN architecture is fully par…
Tensor factorization is a powerful tool to analyse multi-way data. Compared with traditional multi-linear methods, nonlinear tensor factorization models are capable of capturing more complex relationships in the data. However, they are computationally expensive and may suffer severe learning bias in case of extreme dat…
In this paper we introduce a new algorithm for American Monte Carlo that can be used either for American-style options, callable structured products or for computing counterparty credit risk (e.g. CVA or PFE computation). Leveraging least squares regressions, the main novel feature of our algorithm is that it can be fu…
Introduces a natural parallel translation for navigation data.