CryptoNAS improves PI accuracy by 3.4% with 2.4x less latency.
problem Private inference on machine learning models with limited latency.
method Developed CryptoNAS, a novel NAS method for finding models that maximize accuracy within a ReLU budget.
result Improves accuracy by 3.4% and latency by 2.4x over state-of-the-art methods.
PHAZE framework uses zkML and hashing for fast, verifiable LHC trigger decisions.
problem Inefficient inference on large machine learning models for LHC trigger performance.
method Cryptographic techniques like hashing and zkML for low latency, certifiable inference.
result Achieves nanosecond-order latency for LHC triggers, enabling dynamic low-level triggers.
MASnet enhances speech on mobile devices with low latency.
problem Efficiently enhancing speech on mobile devices with low latency.
method MASnet processes linear-scale spectrograms, using ratio masks to enhance noisy frames, and operates in low-latency incremental inference mode.
result MASnet achieves efficient speech enhancement with low latency, reducing FMA/s operations.
DCT-SNN uses DCT to reduce inference latency in SNNs.
problem High inference latency in SNNs.
method Proposes a time-based encoding scheme using DCT to reduce timesteps.
result Achieves top-1 accuracy comparable to standard deep learning while reducing inference latency.
When applying machine learning to sensitive data, one has to find a balance between accuracy, information security, and computational-complexity. Recent studies combined Homomorphic Encryption with neural networks to make inferences while protecting against information leakage. However, these methods are limited by the…
PolyLUT uses polynomials to reduce FPGA latency.
problem Reducing latency in FPGA-based neural network inference.
method Training neural networks using multivariate polynomials as basic building blocks.
result Achieved significant latency and area improvements.
Reducing the latency variance in machine learning inference is a key requirement in many applications. Variance is harder to control in a cloud deployment in the presence of stragglers. In spite of this challenge, inference is increasingly being done in the cloud, due to the advent of affordable machine learning as a s…
This paper proposes an EM approach to reduce inference latency in NAR sequence generation.
problem High inference latency in NAR models due to multi-modality in sequence generation.
method A unified EM framework that jointly optimizes AR and NAR models, with iterative refinement.
result The proposed approach achieves competitive performance with existing NAR models and significantly reduces inference latency.
Automated tool reduces FPGA inference latency to 5 μs for deep neural networks.
problem Deploying ultra low-latency, low-power deep neural networks on FPGAs.
method Extending hls4ml library, using model compression techniques like pruning and quantization-aware training.
result Achieved inference latency of 5 μs with 97% resource reduction.
MLaaS (ML-as-a-Service) offerings by cloud computing platforms are becoming increasingly popular. Hosting pre-trained machine learning models in the cloud enables elastic scalability as the demand grows. But providing low latency and reducing the latency variance is a key requirement. Variance is harder to control in a…
Paper presents FPGA implementation for efficient recurrent neural networks.
problem Implementing recurrent neural networks on FPGAs for low latency.
method Developed hls4ml framework to implement LSTM and GRU layers.
result Demonstrated effective designs for both small and large models.
T2FSNN improves deep SNNs by reducing spikes and latency.
problem Inefficiency in spiking neural networks due to lack of scalable training algorithms.
method Introduces time-to-first-spike coding with kernel-based dynamic threshold and dendrite, combined with gradient-based optimization and early firing methods.
result Reduces inference latency and spike count by 22% and less than 1% compared to burst coding.
DIET-SNN optimizes SNNs for faster, lower-energy image classification.
problem High inference latency and inefficient input encoding in SNNs.
method End-to-end backpropagation to optimize membrane leak and firing threshold.
result Achieves top-1 accuracy of 69% on ImageNet with 5 timesteps and 12x less compute energy.
A power-law fit to the empirical inference-compute frontier in LOB prediction suggests a scaling-law-style frontier.
problem Limit order book prediction
method Using a suite of models ranging from small decision trees to neural LOB architectures
result A power-law fit to the low- and mid-compute non-MLPLOB frontier extrapolates across multiple orders of magnitude and attains R2=0.941 on the excluded high-compute MLPLOB target frontier. A new method reduces inference cost for FwFM by allowing it to scale with item fields only.
problem High computational cost in FwFM for large field counts.
method Low-rank diagonal plus symmetric decomposition for field-wise interactions.
result Aggressive rank reduction outperforms pruning in accuracy and speed.
Recent results at the Large Hadron Collider (LHC) have pointed to enhanced physics capabilities through the improvement of the real-time event processing techniques. Machine learning methods are ubiquitous and have proven to be very powerful in LHC physics, and particle physics as a whole. However, exploration of the u…
Paper proposes new Bayesian neural network models for efficient learning.
problem Efficient learning and model compression in deep neural networks.
method Proposes Spike-and-Slab Group Lasso (SS-GL) and Spike-and-Slab Group Horseshoe (SS-GHS) priors for structured sparsity in Bayesian neural networks.
result Establishes competitive performance in prediction accuracy, model compression, and inference latency compared to baseline models.
This paper evaluates quantization techniques for deep learning inference.
problem Reducing the size and improving inference of deep neural networks.
method Review and empirical evaluation of quantization parameters for various neural network models.
result 8-bit quantization maintains accuracy within 1% of floating-point models.
This paper optimizes caching and model multiplexing for large model inference.
problem Resource consumption and latency challenges in large model deployment.
method Jointly optimizing a caching algorithm (GDSF or LEC) and a model multiplexer for large model inference.
result Achieves optimal rates in offline and online settings with up to 50x improvement over baseline.
Two new methods reduce random forest latency and improve accuracy.
problem High latency and memory demands in deep random forest models.
method DiNo and RanBu convert shallow random forests into efficient predictors.
result RanBu matches or exceeds full-depth random forest accuracy with up to 95% reduction in time.
Power-SMC reduces inference latency for training-free LLM reasoning.
problem Training-free LLM reasoning with low latency.
method Power-SMC, a training-free Sequential Monte Carlo scheme targeting sequence-level power distribution.
result Power-SMC reduces inference latency from 16-28× to 1.4-3.3× over baseline decoding.
TinyML models detect RF and cyber threats in spacecraft with low latency.
problem Detecting cyber-RF threats in autonomous spacecraft with low latency.
method Analysis of classical models (RF, LR, SVM, MLP) for latency-accuracy trade-offs.
result Logistic Regression achieves microsecond-level inference with minimal accuracy loss.
New algorithm reduces costs and latency for large language model inference.
problem Optimizing inference costs and latency for large language models with GPU constraints.
method Formulated as an online scheduling problem with endogenous memory growth, introduced fluid model and WAIT algorithms.
result Reduced costs and latency, especially in near-overloaded and overloaded regimes.
This paper presents a fast Bayesian filtering technique for state estimation.
problem Bottleneck in Bayesian inference for state estimation from noisy sensor data.
method Processor-native uncertainty tracking for uncertainty propagation and inference.
result Deterministic approximate filtering with up to 805x speedup and competitive accuracy.
EQ-Net combines LLR estimation and quantization using deep learning.
problem Unified solution for LLR estimation and quantization.
method Two-stage algorithm using LLR compression as a pretext task.
result Achieves state-of-the-art results with gains in efficiency and latency.
Efficiently models event-based data with general parametric kernels.
problem Inference for Hawkes processes with general parametric kernels requires large datasets.
method Developed a fast ℓ2 gradient-based solver using a discretized version of events. result Improved estimation of pattern latency in brain signals.
New attacks exploit neural network energy and latency, increasing costs by 10-200x.
problem Energy and latency vulnerabilities in neural networks.
method Crafted sponge examples to maximize energy and latency.
result Adversaries can increase energy consumption by 10-200x and delay decisions.
Over the last few years, Deep Neural Networks (DNNs) have become ubiquitous owing to their high accuracy on real-world tasks. However, this increase in accuracy comes at the cost of computationally expensive models leading to higher prediction latencies. Prior efforts to reduce this latency such as quantization, model …
Hyperflux models pruning as a system to reveal weight importance.
problem Pruning large neural networks to reduce latency and power consumption.
method Introduces Hyperflux, a novel L0 method that models pruning as flux and pressure. result Achieves competitive results with ResNet-50, VGG-19, and DeiT-T/S on various datasets.
GOBO compresses 99.9% of BERT model parameters to 3 bits, improving inference efficiency.
problem Efficient execution of attention-based NLP models, especially in terms of latency and energy consumption.
method GOBO quantizes 32-bit floating-point parameters to 3 bits without fine-tuning, using hardware compression and co-designed architectures.
result GOBO maintains model accuracy while significantly reducing inference latency and energy consumption.
Can we automatically design a Convolutional Network (ConvNet) with the highest image classification accuracy under the latency constraint of a mobile device? Neural Architecture Search (NAS) for ConvNet design is a challenging problem due to the combinatorially large design space and search time (at least 200 GPU-hours…
NeuraLUT maps neural networks to lookup tables, reducing latency and improving expressivity.
problem Reducing latency in deep neural networks for FPGA accelerators.
method Mapping entire sub-networks to a single lookup table, introducing skip connections.
result Up to 4.3x lower latency for the same accuracy.
Autoregressive sequence models achieve state-of-the-art performance in domains like machine translation. However, due to the autoregressive factorization nature, these models suffer from heavy latency during inference. Recently, non-autoregressive sequence models were proposed to reduce the inference time. However, the…
A blockchain-based federated learning system with latency analysis.
problem Latency issues in decentralized federated learning architectures.
method Introduced a consortium blockchain and a latency model to analyze the workflow.
result The latency model accurately quantifies actual delays in the system.
Network quantization is one of the most hardware friendly techniques to enable the deployment of convolutional neural networks (CNNs) on low-power mobile devices. Recent network quantization techniques quantize each weight kernel in a convolutional layer independently for higher inference accuracy, since the weight ker…
Galen algorithm compresses neural networks for specific hardware with reduced latency.
problem Finding optimal compression policies for neural networks on specific hardware.
method Reinforcement learning using pruning and quantization to optimize inference latency.
result Compressed ResNet18 for ARM processor reduced inference latency by 80%.
HOLMES improves real-time model serving for ICU patients, balancing accuracy and speed.
problem Real-time model serving is crucial in ICU due to urgency and cost.
method Online model ensemble serving framework for healthcare applications.
result HOLMES achieves high accuracy and sub-second latency for model ensemble serving.
Conventional Neural Architecture Search (NAS) aims at finding a single architecture that achieves the best performance, which usually optimizes task related learning objectives such as accuracy. However, a single architecture may not be representative enough for the whole dataset with high diversity and variety. Intuit…
Optimizes latency and false alarm probability in change detection problems.
problem Balancing latency and false alarms in non-stationary environments.
method Develops order-optimal change detectors under specified latency and false alarm levels.
result Derives a universal lower bound on latency and develops order-optimal detectors.
Study validates low latency's impact on trading profits.
problem Determining the impact of low latency on trading profits.
method Agent-based simulation of trading strategies in a controlled environment.
result Latency inversely affects trading profits; latency rank is key.
This paper reviews methods to create compact neural networks for IoT applications.
problem Complex deep neural networks are costly and slow, hindering real-world deployment.
method Automatic synthesis of compact, accurate DNN/LSTM models.
result Compact neural networks reduce energy consumption, memory, and inference time.
Research examines how strategic latency manipulation impacts Ethereum's network efficiency and decentralization.
problem Impact of artificial latency on Ethereum's network efficiency and decentralization.
method Comprehensive analysis of MEV-Boost auction system and empirical validation with a pilot.
result Increased profitability for node operators and significant systemic challenges like heightened network inefficiencies and centralization risks.
Due to the unparallelizable nature of the autoregressive factorization, AutoRegressive Translation (ART) models have to generate tokens sequentially during decoding and thus suffer from high inference latency. Non-AutoRegressive Translation (NART) models were proposed to reduce the inference time, but could only achiev…
The rising popularity of intelligent mobile devices and the daunting computational cost of deep learning-based models call for efficient and accurate on-device inference schemes. We propose a quantization scheme that allows inference to be carried out using integer-only arithmetic, which can be implemented more efficie…
The Tick library simulates and learns Hawkes processes with latency effects.
problem Correctly modeling causality in order book events with latency.
method Exponential kernels shifted by latency, derived log-likelihood expressions.
result Latency determines most decays in real data, showing decay relationships.
Research optimizes C++ patterns for HFT, reducing latency and improving profitability.
problem Optimizing latency-critical code for high-frequency trading systems.
method Creation of a Low-Latency Programming Repository, optimisation of trading strategy, implementation of Disruptor pattern.
result Significant performance improvements in speed and profitability.
The recent success of Deep Neural Networks (DNNs) has drastically improved the state of the art for many application domains. While achieving high accuracy performance, deploying state-of-the-art DNNs is a challenge since they typically require billions of expensive arithmetic computations. In addition, DNNs are typica…
This paper studies optimal market making for large-tick assets in the presence of latency. We consider a random walk model for the asset price, and formulate the market maker's optimization problem using Markov Decision Processes (MDP). We characterize the value of an order and show that it plays the role of one-period…