Training deep learning models is compute-intensive and there is an industry-wide trend towards hardware specialization to improve performance. To systematically benchmark deep learning platforms, we introduce ParaDnn, a parameterized benchmark suite for deep learning that generates end-to-end models for fully connected…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “GPU/TPU”
Monte Carlo methods are critical to many routines in quantitative finance such as derivatives pricing, hedging and risk metrics. Unfortunately, Monte Carlo methods are very computationally expensive when it comes to running simulations in high-dimensional state spaces where they are still a method of choice in the fina…
We propose an sorting algorithm by Machine Learning method, which shows a huge potential sorting big data. This sorting algorithm can be applied to parallel sorting and is suitable for GPU or TPU acceleration. Furthermore, we discuss the application of this algorithm to sparse hash table.
Efficiently trains BERT on academic GPUs in 12 days.
problem Training large-scale BERT models is expensive and time-consuming.
method Optimizes training on multiple GPUs and nodes, reducing costs.
result Trains BERT on academic GPUs in 12 days, not requiring expensive hardware.
CWY Parametrization: a Solution for Parallelized Optimization of Orthogonal and Stiefel Matricescs.LG
Efficiently optimizes orthogonal and Stiefel matrices on parallel units.
problem Optimization over orthogonal groups on parallel units.
method CWY and T-CWY transforms for parametrization and optimization.
result CWY and T-CWY methods lead to convergence on parallel units.
The impact of the maximally possible batch size (for the better runtime) on performance of graphic processing units (GPU) and tensor processing units (TPU) during training and inference phases is investigated. The numerous runs of the selected deep neural network (DNN) were performed on the standard MNIST and Fashion-M…
Training machine learning (ML) models on large datasets requires considerable computing power. To speed up training, it is typical to distribute training across several machines, often with specialized hardware like GPUs or TPUs. Managing a distributed training job is complex and requires dealing with resource contenti…
NS-RGS improves orthogonal group synchronization with faster convergence.
problem Orthogonal group synchronization from pairwise measurements.
method Newton-Schulz iteration for Riemannian gradient optimization.
result NS-RGS achieves linear convergence and near-optimal accuracy.
By introducing several improvements to the AlphaZero process and architecture, we greatly accelerate self-play learning in Go, achieving a 50x reduction in computation over comparable methods. Like AlphaZero and replications such as ELF OpenGo and Leela Zero, our bot KataGo only learns from neural-net-guided Monte Carl…
AdaNet is a lightweight TensorFlow-based (Abadi et al., 2015) framework for automatically learning high-quality ensembles with minimal expert intervention. Our framework is inspired by the AdaNet algorithm (Cortes et al., 2017) which learns the structure of a neural network as an ensemble of subnetworks. We designed it…
Google's Cloud TPUs are a promising new hardware architecture for machine learning workloads. They have powered many of Google's milestone machine learning achievements in recent years. Google has now made TPUs available for general use on their cloud platform and as of very recently has opened them up further to allow…
Optimized neural networks for Edge TPU achieve high accuracy in real-time image classification.
problem Designing neural networks for hardware accelerators to achieve optimal performance.
method Hardware-aware neural architecture search and model customization for Edge TPU.
result Improved accuracy-latency tradeoff on Pixel 4's Edge TPU compared to existing models.
RETINA Benchmark evaluates Bayesian deep learning on diabetic retinopathy detection.
problem Reliable uncertainty quantification for deep learning models in medical applications.
method Design and evaluation of a real-world diabetic retinopathy dataset and tasks.
result Benchmarking of Bayesian deep learning methods on diabetic retinopathy detection tasks.
We describe TF-Replicator, a framework for distributed machine learning designed for DeepMind researchers and implemented as an abstraction over TensorFlow. TF-Replicator simplifies writing data-parallel and model-parallel research code. The same models can be effortlessly deployed to different cluster architectures (i…
Neural Tangents is a library designed to enable research into infinite-width neural networks. It provides a high-level API for specifying complex and hierarchical neural network architectures. These networks can then be trained and evaluated either at finite-width as usual or in their infinite-width limit. Infinite-wid…
New attacks exploit neural network energy and latency, increasing costs by 10-200x.
problem Energy and latency vulnerabilities in neural networks.
method Crafted sponge examples to maximize energy and latency.
result Adversaries can increase energy consumption by 10-200x and delay decisions.
BlackJAX simplifies Bayesian inference with modular, fast implementations.
problem Efficient and modular Bayesian inference methods for complex models.
method Functional approach to implementing sampling and variational inference algorithms in JAX.
result Ease of use, speed, and modularity in implementing Bayesian inference.
Trade uncertainty impact on stock-bond correlations: Insights from conditional correlation modelsq-fin.ST
The study shows how trade uncertainty affects stock-bond correlations over time.
problem Impact of trade policy uncertainty on stock-bond correlations.
method Daily data analysis using GARCH-based models (CCC, STCC, DCC) with TPU and political dummy variables.
result Time-varying correlation models better capture the dynamics of stock-bond correlations than constant models.
The paper models and analyzes faults in TPU-based neural networks.
problem Faults in TPU matrix multiplication units cause errors in DNN inference.
method Formal model using DTMC, probabilistic model checking, and experiments.
result Classification accuracy is sensitive to fault type, location, and neural network layers.
Can we automatically design a Convolutional Network (ConvNet) with the highest image classification accuracy under the latency constraint of a mobile device? Neural Architecture Search (NAS) for ConvNet design is a challenging problem due to the combinatorially large design space and search time (at least 200 GPU-hours…
Neural network quantization is becoming an industry standard to efficiently deploy deep learning models on hardware platforms, such as CPU, GPU, TPU, and FPGAs. However, we observe that the conventional quantization approaches are vulnerable to adversarial attacks. This paper aims to raise people's awareness about the …
Large, pre-trained generative models have been increasingly popular and useful to both the research and wider communities. Specifically, BigGANs a class-conditional Generative Adversarial Networks trained on ImageNet---achieved excellent, state-of-the-art capability in generating realistic photos. However, fine-tuning …
Proposes differentiable and sparse top-k operators for neural networks.
problem Discontinuity of top-k operator makes it unsuitable for end-to-end training with backpropagation.
method Formulates top-k as a linear program over permutahedron, introduces p-norm regularization, and uses isotonic optimization.
result Successfully applied to neural network pruning, fine-tuning, and routing.
In a recent paper, we have demonstrated how the affinity between TPUs and multi-dimensional financial simulation resulted in fast Monte Carlo simulations that could be setup in a few lines of python Tensorflow code. We also presented a major benefit from writing high performance simulations in an automated differentiat…
Framework for precise recall control in spatial conflation tasks.
problem Precise recall control in large-scale spatial conflation tasks to avoid downstream analytics failures and excessive manual review.
method End-to-end framework using equigrid bounding-box filter, CSR representation, neural ranker, and inverse-variance weighted ensemble of threshold estimators.
result Achieves exact recall with sub-percent variance over tens of millions of geometry pairs, runs on a single TPU v3 core.
Can we automatically design a Convolutional Network (ConvNet) with the highest image classification accuracy under the runtime constraint of a mobile device? Neural architecture search (NAS) has revolutionized the design of hardware-efficient ConvNets by automating this process. However, the NAS problem remains challen…
New method uses MCTS only at test time for faster game learning.
problem High computational demands in game learning.
method Combines MCTS with TD learning agents for faster, reproducible agents.
result First learning-from-scratch agent to beat Edax up to level 7.
New GPU algorithm boosts machine learning with larger datasets.
problem Limited GPU memory restricts training data size.
method Out-of-core GPU gradient boosting algorithm.
result Training larger datasets on GPUs without accuracy loss.
Massively parallel architectures such as the GPU are becoming increasingly important due to the recent proliferation of data. In this paper, we propose a key class of hybrid parallel graphlet algorithms that leverages multiple CPUs and GPUs simultaneously for computing k-vertex induced subgraph statistics (called graph…
ProtTrans models predict protein features without evolutionary info.
problem Predicting protein features from amino acid sequences.
method Self-supervised deep learning on large protein datasets.
result ProtT5 embeddings outperform state-of-the-art for per-residue predictions.
Extends GENO framework for GPU optimization of constrained ML problems.
problem Constrained optimization in classical machine learning.
method Extends GENO framework to GPU optimization, specifying problems in a modeling language.
result Solvers on GPU outperform state-of-the-art approaches by several orders of magnitude.
GPU computing has become popular in computational finance and many financial institutions are moving their CPU based applications to the GPU platform. Since most Monte Carlo algorithms are embarrassingly parallel, they benefit greatly from parallel implementations, and consequently Monte Carlo has become a focal point …
GShard enables scaling of large neural networks with automatic sharding and lightweight APIs.
problem Scaling neural networks to handle vast training data and compute efficiently.
method GShard uses lightweight annotation APIs and XLA compiler extensions for parallel computation.
result GShard successfully trained a 600 billion parameter model on 2048 TPUs in 4 days.
We introduce CuLE (CUDA Learning Environment), a CUDA port of the Atari Learning Environment (ALE) which is used for the development of deep reinforcement algorithms. CuLE overcomes many limitations of existing CPU-based emulators and scales naturally to multiple GPUs. It leverages GPU parallelization to run thousands …
GPU speeds up derivatives sensitivity computation for Heston model options.
problem Efficient computation of option Greeks under the Heston model.
method Implemented exact simulation and novel Milstein discretisation methods on GPU.
result GPU speeds up Greeks computation up to 200x compared to CPU methods.
JAXFit speeds up curve fitting on GPUs.
problem Nonlinear least squares curve fitting problems.
method Trust region method on GPU with automatic differentiation.
result Significantly faster than CPU and other GPU libraries.
GPU optimization speeds up large-scale classification tasks.
problem Efficiently training large-scale classification models on GPUs.
method Judecious GPU-optimization principles applied to TRON algorithm.
result Significant speedups for logistic regression and SVM classification.
LAD-BNet improves real-time energy forecasting on edge devices.
problem Real-time energy forecasting on edge devices for smart grid optimization and intelligent buildings.
method Hybrid neural architecture combining temporal lag exploitation and TCN with dilated convolutions.
result 14.49% MAPE at 1-hour horizon with 18ms inference time on Edge TPU, 8-12x faster than CPU.
A GPU framework speeds up BnB for discrete optimization problems.
problem Optimizing large-scale discrete problems with GPU limitations.
method Parallel BnB nodes in GPU batches, using padding and custom kernels.
result One to two orders of magnitude speedup and zero optimality gap.
New asynchronous SGD algorithms achieve optimal performance in distributed learning.
problem Asynchronous training introduces staleness, complicating optimization analysis.
method Developed rigorous framework for asynchronous first order stochastic optimization.
result Asynchronous SGD can achieve optimal time complexity, matching synchronous methods.
We describe the multi-GPU gradient boosting algorithm implemented in the XGBoost library (https://github.com/dmlc/xgboost). Our algorithm allows fast, scalable training on multi-GPU systems with all of the features of the XGBoost library. We employ data compression techniques to minimise the usage of scarce GPU memory …
Can we reduce the search cost of Neural Architecture Search (NAS) from days down to only few hours? NAS methods automate the design of Convolutional Networks (ConvNets) under hardware constraints and they have emerged as key components of AutoML frameworks. However, the NAS problem remains challenging due to the combin…
Speeds up deep neural networks training by 10x using GPU concurrency.
problem Training deep residual neural networks efficiently.
method Layer-wise parallel training with GPU concurrency and Nonlinear Multigrid.
result 10.2x speedup over traditional techniques.
In this paper, we present a novel massively parallel algorithm for accelerating the decision tree building procedure on GPUs (Graphics Processing Units), which is a crucial step in Gradient Boosted Decision Tree (GBDT) and random forests training. Previous GPU based tree building algorithms are based on parallel multi-…
This paper optimizes deep learning training by efficiently sharding weight updates across replicas.
problem Redundant weight update computation on all replicas in data-parallel training.
method Automatic sharding of weight updates using static analysis and transformations on the training graph.
result Substantial speedups achieved on large-scale models using Cloud TPUs.
Deep neural networks have yielded superior performance in many applications; however, the gradient computation in a deep model with millions of instances lead to a lengthy training process even with modern GPU/TPU hardware acceleration. In this paper, we propose AutoAssist, a simple framework to accelerate training of …
RFX accelerates and compresses Random Forests for large datasets.
problem Memory bottleneck in proximity matrices limits Random Forest analysis.
method QLORA compression, CPU TriBlock storage, GPU batch sizing, 3D MDS visualization.
result Proximity-based Random Forest analysis on larger datasets is feasible.
UMAP speeds up significantly with GPU acceleration.
problem Efficient GPU implementation of UMAP.
method Various techniques to optimize UMAP for GPU, including data partitioning and parallelization.
result Up to 100x speedup in practice.