Automated tool reduces FPGA inference latency to 5 μs for deep neural networks.
problem Deploying ultra low-latency, low-power deep neural networks on FPGAs.
method Extending hls4ml library, using model compression techniques like pruning and quantization-aware training.
result Achieved inference latency of 5 μs with 97% resource reduction.
Freezing intermediate layers reduces DNN inference latency.
problem High latency in deep learning inference due to computationally expensive models.
method Introduce approximate caching at each intermediate layer to avoid running all layers for many inference requests.
result Reduces the number of effective layers by half for 91.58% of CIFAR-10 requests.
Exchanges implement intentional trade delays to limit the harmful impact of low-latency trading. Do such "speed bumps" curb investment in fast trading technology? Data is scarce since trading technologies are proprietary. We build an experimental trading platform where participants face speed bumps and can invest in fa…
This paper proposes an EM approach to reduce inference latency in NAR sequence generation.
problem High inference latency in NAR models due to multi-modality in sequence generation.
method A unified EM framework that jointly optimizes AR and NAR models, with iterative refinement.
result The proposed approach achieves competitive performance with existing NAR models and significantly reduces inference latency.
DCT-SNN uses DCT to reduce inference latency in SNNs.
problem High inference latency in SNNs.
method Proposes a time-based encoding scheme using DCT to reduce timesteps.
result Achieves top-1 accuracy comparable to standard deep learning while reducing inference latency.
Reducing the latency variance in machine learning inference is a key requirement in many applications. Variance is harder to control in a cloud deployment in the presence of stragglers. In spite of this challenge, inference is increasingly being done in the cloud, due to the advent of affordable machine learning as a s…
MLaaS (ML-as-a-Service) offerings by cloud computing platforms are becoming increasingly popular. Hosting pre-trained machine learning models in the cloud enables elastic scalability as the demand grows. But providing low latency and reducing the latency variance is a key requirement. Variance is harder to control in a…
Improves text-to-speech speed by interleaving character reading and audio synthesis.
problem Latency in text-to-speech models limits their use in time-sensitive tasks.
method Reinforcement learning to train an agent to choose the order of character reading and audio synthesis.
result The proposed method successfully balances latency and audio quality.
Real-time semantic segmentation for autonomous vehicles on FPGA reduces latency and power consumption.
problem Efficient real-time semantic segmentation for autonomous vehicles.
method Compressed ENet architecture, FPGA deployment, batch processing, filter reduction, quantization-aware training.
result Reduced latency to 3 ms per image with batch size of ten and 40% resource utilization.
Research optimizes C++ patterns for HFT, reducing latency and improving profitability.
problem Optimizing latency-critical code for high-frequency trading systems.
method Creation of a Low-Latency Programming Repository, optimisation of trading strategy, implementation of Disruptor pattern.
result Significant performance improvements in speed and profitability.
T2FSNN improves deep SNNs by reducing spikes and latency.
problem Inefficiency in spiking neural networks due to lack of scalable training algorithms.
method Introduces time-to-first-spike coding with kernel-based dynamic threshold and dendrite, combined with gradient-based optimization and early firing methods.
result Reduces inference latency and spike count by 22% and less than 1% compared to burst coding.
High frequency trading has led to widespread efforts to reduce information propagation delays between physically distant exchanges. Using relativistically correct millisecond-resolution tick data, we document a 3-millisecond decrease in one-way communication time between the Chicago and New York areas that has occurred…
A DRL approach optimizes resource allocation in BFL to reduce latency and energy consumption.
problem Energy and CPU constraints of mobile devices and increased training latency due to blockchain mining.
method Deep Reinforcement Learning (DRL) to derive optimal decisions for MLMO.
result Optimal resource allocation and block generation rate to minimize system latency, energy consumption, and incentive cost.
We present a systematic analysis on the performance of a phonetic recogniser when the window of input features is not symmetric with respect to the current frame. The recogniser is based on Context Dependent Deep Neural Networks (CD-DNNs) and Hidden Markov Models (HMMs). The objective is to reduce the latency of the sy…
Power-SMC reduces inference latency for training-free LLM reasoning.
problem Training-free LLM reasoning with low latency.
method Power-SMC, a training-free Sequential Monte Carlo scheme targeting sequence-level power distribution.
result Power-SMC reduces inference latency from 16-28× to 1.4-3.3× over baseline decoding.
DIET-SNN optimizes SNNs for faster, lower-energy image classification.
problem High inference latency and inefficient input encoding in SNNs.
method End-to-end backpropagation to optimize membrane leak and firing threshold.
result Achieves top-1 accuracy of 69% on ImageNet with 5 timesteps and 12x less compute energy.
Density-Softmax improves uncertainty estimation and robustness without sampling, reducing model size and latency.
problem Sampling-based uncertainty estimation methods suffer from large model size and high latency.
method Combines a Lipschitz-constrained feature extractor with the softmax layer to create a sampling-free deterministic framework.
result Density-Softmax reduces over-confidence under distribution shifts and achieves competitive results in uncertainty and robustness.
PolyLUT uses polynomials to reduce FPGA latency.
problem Reducing latency in FPGA-based neural network inference.
method Training neural networks using multivariate polynomials as basic building blocks.
result Achieved significant latency and area improvements.
Two new methods reduce random forest latency and improve accuracy.
problem High latency and memory demands in deep random forest models.
method DiNo and RanBu convert shallow random forests into efficient predictors.
result RanBu matches or exceeds full-depth random forest accuracy with up to 95% reduction in time.
EQ-Net combines LLR estimation and quantization using deep learning.
problem Unified solution for LLR estimation and quantization.
method Two-stage algorithm using LLR compression as a pretext task.
result Achieves state-of-the-art results with gains in efficiency and latency.
TinyLSTMs reduces speech enhancement model size and latency for hearing aids.
problem Large RNNs limit practical deployment in hearing aid hardware.
method Model compression techniques (pruning, integer quantization, state update skipping) for RNN speech enhancement.
result Reduction in model size and operations by 11.9imes and 2.9imes, respectively, without perceptual degradation. Prophet predicts device qualities for FL to reduce training latency.
problem Bad candidate-selection leads to large training and reporting latency in FL.
method Each device predicts its own training and reporting phases using LSTM. The algorithm is implemented with DRL.
result The proposed approach outperforms reactive algorithms in real-world experiments.
S3NAS finds high-accuracy CNN architectures for NPUs in 3 hours.
problem Finding optimal CNN architectures for NPUs with high accuracy and low latency.
method Supernet design, Single-Path NAS, scaling, analytical latency estimation.
result 82.72% top-1 accuracy on ImageNet with 11.66 ms latency.
Interactive applications incorporating high-data rate sensing and computer vision are becoming possible due to novel runtime systems and the use of parallel computation resources. To allow interactive use, such applications require careful tuning of multiple application parameters to meet required fidelity and latency …
Long short-term memory (LSTM) has been widely used for sequential data modeling. Researchers have increased LSTM depth by stacking LSTM cells to improve performance. This incurs model redundancy, increases run-time delay, and makes the LSTMs more prone to overfitting. To address these problems, we propose a hidden-laye…
Paper proposes new Bayesian neural network models for efficient learning.
problem Efficient learning and model compression in deep neural networks.
method Proposes Spike-and-Slab Group Lasso (SS-GL) and Spike-and-Slab Group Horseshoe (SS-GHS) priors for structured sparsity in Bayesian neural networks.
result Establishes competitive performance in prediction accuracy, model compression, and inference latency compared to baseline models.
GOBO compresses 99.9% of BERT model parameters to 3 bits, improving inference efficiency.
problem Efficient execution of attention-based NLP models, especially in terms of latency and energy consumption.
method GOBO quantizes 32-bit floating-point parameters to 3 bits without fine-tuning, using hardware compression and co-designed architectures.
result GOBO maintains model accuracy while significantly reducing inference latency and energy consumption.
Machine Learning models are often composed of pipelines of transformations. While this design allows to efficiently execute single model components at training time, prediction serving has different requirements such as low latency, high throughput and graceful performance degradation under heavy load. Current predicti…
NeuraLUT maps neural networks to lookup tables, reducing latency and improving expressivity.
problem Reducing latency in deep neural networks for FPGA accelerators.
method Mapping entire sub-networks to a single lookup table, introducing skip connections.
result Up to 4.3x lower latency for the same accuracy.
SpeakerStew verifies 46 languages with reduced training and inference costs.
problem Speaker verification for 46 languages with smart speaker interactions.
method Pooling multilingual data, triage between text-dependent and text-independent models.
result Training on multiple languages generalizes well and reduces computational requirements.
A blockchain-based federated learning system with latency analysis.
problem Latency issues in decentralized federated learning architectures.
method Introduced a consortium blockchain and a latency model to analyze the workflow.
result The latency model accurately quantifies actual delays in the system.
Select-DC reduces GFLOPS for uncertainty estimation in neural networks.
problem Computational inefficiency in estimating model uncertainty for low-latency applications.
method Select-DC uses a subset of layers to model epistemic uncertainty with MCDC, reducing GFLOPS.
result Significant reduction in GFLOPS required for uncertainty estimation with marginal performance loss.
LanguaShrink compresses prompts using psycholinguistic principles to reduce costs.
problem Reducing computational cost and efficiency in large language model inference.
method Leverages psycholinguistic principles and the Ebbinghaus memory curve to compress prompts.
result Achieves up to 26 times compression while maintaining semantic similarity.
Optimizes latency and false alarm probability in change detection problems.
problem Balancing latency and false alarms in non-stationary environments.
method Develops order-optimal change detectors under specified latency and false alarm levels.
result Derives a universal lower bound on latency and develops order-optimal detectors.
New algorithm reduces costs and latency for large language model inference.
problem Optimizing inference costs and latency for large language models with GPU constraints.
method Formulated as an online scheduling problem with endogenous memory growth, introduced fluid model and WAIT algorithms.
result Reduced costs and latency, especially in near-overloaded and overloaded regimes.
We are interested in parallelizing the Least Angle Regression (LARS) algorithm for fitting linear regression models to high-dimensional data. We consider two parallel and communication avoiding versions of the basic LARS algorithm. The two algorithms have different asymptotic costs and practical performance. One offers…
A new method reduces inference cost for FwFM by allowing it to scale with item fields only.
problem High computational cost in FwFM for large field counts.
method Low-rank diagonal plus symmetric decomposition for field-wise interactions.
result Aggressive rank reduction outperforms pruning in accuracy and speed.
Study validates low latency's impact on trading profits.
problem Determining the impact of low latency on trading profits.
method Agent-based simulation of trading strategies in a controlled environment.
result Latency inversely affects trading profits; latency rank is key.
We address the challenging problem of efficient inference across many devices and resource constraints, especially on edge devices. Conventional approaches either manually design or use neural architecture search (NAS) to find a specialized neural network and train it from scratch for each case, which is computationall…
Research examines how strategic latency manipulation impacts Ethereum's network efficiency and decentralization.
problem Impact of artificial latency on Ethereum's network efficiency and decentralization.
method Comprehensive analysis of MEV-Boost auction system and empirical validation with a pilot.
result Increased profitability for node operators and significant systemic challenges like heightened network inefficiencies and centralization risks.
This paper evaluates quantization techniques for deep learning inference.
problem Reducing the size and improving inference of deep neural networks.
method Review and empirical evaluation of quantization parameters for various neural network models.
result 8-bit quantization maintains accuracy within 1% of floating-point models.
Hyperflux models pruning as a system to reveal weight importance.
problem Pruning large neural networks to reduce latency and power consumption.
method Introduces Hyperflux, a novel L0 method that models pruning as flux and pressure. result Achieves competitive results with ResNet-50, VGG-19, and DeiT-T/S on various datasets.
The Tick library simulates and learns Hawkes processes with latency effects.
problem Correctly modeling causality in order book events with latency.
method Exponential kernels shifted by latency, derived log-likelihood expressions.
result Latency determines most decays in real data, showing decay relationships.
MICO uses mutual information co-training to improve selective search efficiency.
problem Efficiently search and route unseen queries in large-scale search systems.
method Mutual Information Co-training framework for selective search with minimal supervision.
result Significantly improves performance on multiple metrics of selective search.
We study collaborative machine learning (ML) across wireless devices, each with its own local dataset. Offloading these datasets to a cloud or an edge server to implement powerful ML solutions is often not feasible due to latency, bandwidth and privacy constraints. Instead, we consider federated edge learning (FEEL), w…
MASnet enhances speech on mobile devices with low latency.
problem Efficiently enhancing speech on mobile devices with low latency.
method MASnet processes linear-scale spectrograms, using ratio masks to enhance noisy frames, and operates in low-latency incremental inference mode.
result MASnet achieves efficient speech enhancement with low latency, reducing FMA/s operations.
CryptoNAS improves PI accuracy by 3.4% with 2.4x less latency.
problem Private inference on machine learning models with limited latency.
method Developed CryptoNAS, a novel NAS method for finding models that maximize accuracy within a ReLU budget.
result Improves accuracy by 3.4% and latency by 2.4x over state-of-the-art methods.
PHAZE framework uses zkML and hashing for fast, verifiable LHC trigger decisions.
problem Inefficient inference on large machine learning models for LHC trigger performance.
method Cryptographic techniques like hashing and zkML for low latency, certifiable inference.
result Achieves nanosecond-order latency for LHC triggers, enabling dynamic low-level triggers.