Julia programs and ML models are compiled and offloaded to Google Cloud TPUs.
problem Efficiently compiling and executing machine learning models on Google Cloud TPUs.
method Developed a method to compile and offload Julia programs and ML models to Google Cloud TPUs using the Google XLA compiler.
result Successfully fused a VGG19 model into a single TPU executable, achieving 0.23s evaluation time on a batch of 100 images.
Benchmarking TPU, GPU, and CPU for deep learning models.
problem Improving performance in deep learning training.
method ParaDnn benchmark suite for FC, CNN, and RNN models on TPU, GPU, and CPU.
result TPU, GPU, and CPU have unique strengths for different types of models.
TPUs speed up financial Monte Carlo simulations.
problem High computational cost of Monte Carlo simulations in finance.
method Empirical experiments comparing TPUs to GPUs for financial Monte Carlo tasks.
result TPUs provide accurate and fast estimators for financial Monte Carlo tasks.
Optimized neural networks for Edge TPU achieve high accuracy in real-time image classification.
problem Designing neural networks for hardware accelerators to achieve optimal performance.
method Hardware-aware neural architecture search and model customization for Edge TPU.
result Improved accuracy-latency tradeoff on Pixel 4's Edge TPU compared to existing models.
The study shows how trade uncertainty affects stock-bond correlations over time.
problem Impact of trade policy uncertainty on stock-bond correlations.
method Daily data analysis using GARCH-based models (CCC, STCC, DCC) with TPU and political dummy variables.
result Time-varying correlation models better capture the dynamics of stock-bond correlations than constant models.
The paper models and analyzes faults in TPU-based neural networks.
problem Faults in TPU matrix multiplication units cause errors in DNN inference.
method Formal model using DTMC, probabilistic model checking, and experiments.
result Classification accuracy is sensitive to fault type, location, and neural network layers.
Investigates the impact of batch size on GPU and TPU performance.
problem Optimizing performance of GPUs and TPUs during training and inference phases.
method Investigated the impact of batch size on performance of GPUs and TPUs using standard MNIST and Fashion-MNIST datasets.
result Significant speedup was achieved even with low-scale usage of TPUv2 units, up to 10x for training and 2x for prediction.
Paper discusses optimizing deep learning models for supercomputers.
problem Training deep learning models at high petaFLOPS scale.
method Three systems-related optimizations: distributed batch normalization, input pipeline, and 2-D torus all-reduce.
result Trained ResNet-50 on ImageNet to 76.3% accuracy in 2.2 minutes.
Framework for precise recall control in spatial conflation tasks.
problem Precise recall control in large-scale spatial conflation tasks to avoid downstream analytics failures and excessive manual review.
method End-to-end framework using equigrid bounding-box filter, CSR representation, neural ranker, and inverse-variance weighted ensemble of threshold estimators.
result Achieves exact recall with sub-percent variance over tens of millions of geometry pairs, runs on a single TPU v3 core.
We propose an O(N⋅M) sorting algorithm by Machine Learning method, which shows a huge potential sorting big data. This sorting algorithm can be applied to parallel sorting and is suitable for GPU or TPU acceleration. Furthermore, we discuss the application of this algorithm to sparse hash table.
This paper improves financial simulations using Tensor Processing Units and Tensorflow.
problem Estimating sensitivities in financial models efficiently.
method Utilizing Tensor Processing Units and Tensorflow for fast and automated differentiation.
result Single line of code for estimating sensitivities in financial models.
Efficiently optimizes orthogonal and Stiefel matrices on parallel units.
problem Optimization over orthogonal groups on parallel units.
method CWY and T-CWY transforms for parametrization and optimization.
result CWY and T-CWY methods lead to convergence on parallel units.
GShard enables scaling of large neural networks with automatic sharding and lightweight APIs.
problem Scaling neural networks to handle vast training data and compute efficiently.
method GShard uses lightweight annotation APIs and XLA compiler extensions for parallel computation.
result GShard successfully trained a 600 billion parameter model on 2048 TPUs in 4 days.
LAD-BNet improves real-time energy forecasting on edge devices.
problem Real-time energy forecasting on edge devices for smart grid optimization and intelligent buildings.
method Hybrid neural architecture combining temporal lag exploitation and TCN with dilated convolutions.
result 14.49% MAPE at 1-hour horizon with 18ms inference time on Edge TPU, 8-12x faster than CPU.
This paper optimizes deep learning training by efficiently sharding weight updates across replicas.
problem Redundant weight update computation on all replicas in data-parallel training.
method Automatic sharding of weight updates using static analysis and transformations on the training graph.
result Substantial speedups achieved on large-scale models using Cloud TPUs.
Deep neural networks (DNN) are increasingly being accelerated on application-specific hardware such as the Google TPU designed especially for deep learning. Timing speculation is a promising approach to further increase the energy efficiency of DNN accelerators. Architectural exploration for timing speculation requires…
NS-RGS improves orthogonal group synchronization with faster convergence.
problem Orthogonal group synchronization from pairwise measurements.
method Newton-Schulz iteration for Riemannian gradient optimization.
result NS-RGS achieves linear convergence and near-optimal accuracy.
TonY simplifies distributed ML job management.
problem Managing distributed ML jobs is complex and resource-intensive.
method TonY is an open-source orchestrator for distributed ML jobs.
result TonY simplifies distributed ML job management.
RETINA Benchmark evaluates Bayesian deep learning on diabetic retinopathy detection.
problem Reliable uncertainty quantification for deep learning models in medical applications.
method Design and evaluation of a real-world diabetic retinopathy dataset and tasks.
result Benchmarking of Bayesian deep learning methods on diabetic retinopathy detection tasks.
Mesh-TensorFlow enables efficient deep learning on large clusters.
problem Memory constraints and inefficiency in batch-splitting for large models.
method Introduces Mesh-TensorFlow for specifying general tensor computations across a multi-dimensional mesh of processors.
result Trains Transformer models with up to 5 billion parameters on TPU meshes of up to 512 cores.
A cost-effective method to generate high-resolution images using wavelet-based super-resolution.
problem High compute resources required for training state-of-the-art image generation models.
method Wavelet-based two-step training framework: low-frequency sampling followed by super-resolution.
result Achieves FID of 10.59 on ImageNet 512x512 with half the compute of BigGAN.
Efficiently trains BERT on academic GPUs in 12 days.
problem Training large-scale BERT models is expensive and time-consuming.
method Optimizes training on multiple GPUs and nodes, reducing costs.
result Trains BERT on academic GPUs in 12 days, not requiring expensive hardware.
AdaNet automates ensemble learning with TensorFlow.
problem Automatic ensemble learning for high-quality models.
method Inspired by AdaNet algorithm, integrates TensorFlow ecosystem, offers flexible API.
result Efficiently accelerates training on various hardware.
Sparse deep neural networks(DNNs) are efficient in both memory and compute when compared to dense DNNs. But due to irregularity in computation of sparse DNNs, their efficiencies are much lower than that of dense DNNs on regular parallel hardware such as TPU. This inefficiency leads to poor/no performance benefits for s…
BlackJAX simplifies Bayesian inference with modular, fast implementations.
problem Efficient and modular Bayesian inference methods for complex models.
method Functional approach to implementing sampling and variational inference algorithms in JAX.
result Ease of use, speed, and modularity in implementing Bayesian inference.
VeLO learns versatile optimizers from deep learning tasks.
problem Training deep learning models with hand-designed optimizers.
method Meta-training a neural network optimizer on a wide variety of optimization tasks.
result The learned optimizer automatically adapts to different optimization tasks without hyperparameter tuning.
Efficient algorithm finds fast Transformer models.
problem Slow inference time of Transformer models.
method Decompose Transformer architecture into components, use sampling-based one-shot search.
result Achieved 10% to 30% speedup on pre-trained BERT and 70% on top of a previous state-of-the-art model.
TF-Replicator simplifies distributed ML for researchers.
problem Complex distributed ML code for researchers.
method Abstraction over TensorFlow, data-parallel and model-parallel.
result Strong scalability without distributed systems expertise.
KataGo accelerates Go self-play learning by 50x.
problem Efficiently learning in large state spaces like Go.
method Improved AlphaZero process and architecture.
result 50x reduction in computation time.
Neural Tangents simplifies infinite-width neural networks for research.
problem Training and studying infinite-width neural networks.
method High-level API for specifying complex architectures, analytical or gradient-based training, and automatic distribution.
result Analytical training of infinite-width networks and automatic parallelization.
Single-Path NAS designs efficient ConvNets for mobile devices in hours.
problem Designing efficient ConvNets for mobile devices under latency constraints.
method Single-Path NAS, a differentiable method that reduces search cost and inference time.
result Achieves state-of-the-art accuracy on ImageNet with 79ms inference latency.
Sparsity helps reduce the computational complexity of deep neural networks by skipping zeros. Taking advantage of sparsity is listed as a high priority in next generation DNN accelerators such as TPU. The structure of sparsity, i.e., the granularity of pruning, affects the efficiency of hardware accelerator design as w…
A new method defends neural networks from adversarial attacks.
problem Adversarial attacks on quantized neural networks.
method Defensive Quantization (DQ) method to control Lipschitz constant.
result DQ method defends neural networks from adversarial attacks and improves accuracy.
New attacks exploit neural network energy and latency, increasing costs by 10-200x.
problem Energy and latency vulnerabilities in neural networks.
method Crafted sponge examples to maximize energy and latency.
result Adversaries can increase energy consumption by 10-200x and delay decisions.
Proposes differentiable and sparse top-k operators for neural networks.
problem Discontinuity of top-k operator makes it unsuitable for end-to-end training with backpropagation.
method Formulates top-k as a linear program over permutahedron, introduces p-norm regularization, and uses isotonic optimization.
result Successfully applied to neural network pruning, fine-tuning, and routing.
EfficientNet improves ConvNets by scaling depth, width, and resolution effectively.
problem Improving ConvNets' accuracy and efficiency at larger scales.
method Uniformly scaling depth, width, and resolution using a compound coefficient.
result EfficientNet-B7 achieves state-of-the-art 84.3% top-1 accuracy on ImageNet.
Users create melodies with AI, harmonized in the style of Bach.
problem Making music composition accessible to non-experts.
method Simplified interface, machine learning model Coconet, optimized for web.
result Users spent 350 years worth of time playing with the Bach Doodle.
New method uses MCTS only at test time for faster game learning.
problem High computational demands in game learning.
method Combines MCTS with TD learning agents for faster, reproducible agents.
result First learning-from-scratch agent to beat Edax up to level 7.
This study shows how trade policy uncertainty affects stock-T bill correlations.
problem The impact of trade policy uncertainty on stock-T bill relationships.
method Extended Dynamic Conditional Correlation (DCC) framework incorporating exogenous variables.
result Trade policy uncertainty significantly alters stock-T bill correlations, especially under specific political conditions.
Cost-effective method improves and re-purposes pre-trained GANs by fine-tuning class-embeddings.
problem Fine-tuning BigGANs from scratch is impractical due to instability and high computational cost.
method Fine-tuning only the class-embedding layer of pre-trained GANs.
result Significantly improved realism and diversity of samples, re-purposed for new tasks, and de-biased or improved diversity.
GOBO compresses 99.9% of BERT model parameters to 3 bits, improving inference efficiency.
problem Efficient execution of attention-based NLP models, especially in terms of latency and energy consumption.
method GOBO quantizes 32-bit floating-point parameters to 3 bits without fine-tuning, using hardware compression and co-designed architectures.
result GOBO maintains model accuracy while significantly reducing inference latency and energy consumption.
Improved ImageNet classification with semi-supervised learning.
problem Image classification with limited labeled data.
method Noisy Student Training: semi-supervised learning with noisy student models.
result 88.4% top-1 accuracy on ImageNet, 2.0% better than state-of-the-art.
ProtTrans models predict protein features without evolutionary info.
problem Predicting protein features from amino acid sequences.
method Self-supervised deep learning on large protein datasets.
result ProtT5 embeddings outperform state-of-the-art for per-residue predictions.
Single-Path NAS designs efficient ConvNets in under 4 hours.
problem Designing efficient ConvNets for mobile devices under time constraints.
method Differentiable NAS method using one single-path ConvNet.
result Achieves 74.96% top-1 accuracy on ImageNet with 79ms latency.
New asynchronous SGD algorithms achieve optimal performance in distributed learning.
problem Asynchronous training introduces staleness, complicating optimization analysis.
method Developed rigorous framework for asynchronous first order stochastic optimization.
result Asynchronous SGD can achieve optimal time complexity, matching synchronous methods.
Improves matrix multiplication throughput for asymmetric bit-width operands.
problem Matrix multiplications between asymmetric bit-width operands, especially 8- and 4-bit, are not efficiently handled by existing SIMD instructions.
method Proposes a new SIMD matrix multiplication instruction that uses mixed precision on inputs (8- and 4-bit) and accumulates into 16-bit output, improving throughput.
result Offers 2x improvement in throughput compared to existing symmetric-operand-size instructions, with negligible overflow.
Single-Path NAS reduces NAS search cost to 3 hours, achieving state-of-the-art mobile image classification.
problem Efficiently automate ConvNet design under mobile latency constraints.
method Single-Path NAS, using one single-path ConvNet with shared parameters.
result Achieves state-of-the-art top-1 ImageNet accuracy (75.62%) in 8 epochs (24 TPU-hours).
AutoAssist accelerates deep neural network training by filtering out less informative instances.
problem Efficiently training deep neural networks with millions of instances.
method AutoAssist uses a shrinking operation to filter out less informative instances, accelerating training.
result AutoAssist reduces training time by 40% for ResNet and 30% for transformer models.