Bayesian classification and regression with high order interactions is largely infeasible because Markov chain Monte Carlo (MCMC) would need to be applied with a great many parameters, whose number increases rapidly with the order. In this paper we show how to make it feasible by effectively reducing the number of para…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “parameter compression”
Wavelets help compress neural networks efficiently.
problem Efficiently compressing linear layers in neural networks.
method Learnable wavelet transforms to compress RNNs.
result Wavelet compressed RNNs have fewer parameters and perform competitively.
Deep Neural Networks (DNNs) are applied in a wide range of usecases. There is an increased demand for deploying DNNs on devices that do not have abundant resources such as memory and computation units. Recently, network compression through a variety of techniques such as pruning and quantization have been proposed to r…
DoCoFL compresses model updates for cross-device federated learning.
problem Downlink compression for cross-device federated learning where clients may appear only once.
method Proposes DoCoFL framework for downlink compression in cross-device federated learning.
result Significant bi-directional bandwidth reduction with competitive accuracy.
Model compression has been introduced to reduce the required hardware resources while maintaining the model accuracy. Lots of techniques for model compression, such as pruning, quantization, and low-rank approximation, have been suggested along with different inference implementation characteristics. Adopting model com…
Galen algorithm compresses neural networks for specific hardware with reduced latency.
problem Finding optimal compression policies for neural networks on specific hardware.
method Reinforcement learning using pruning and quantization to optimize inference latency.
result Compressed ResNet18 for ARM processor reduced inference latency by 80%.
SHVC improves image compression with fewer parameters.
problem Challenges in VAE compression, especially with bits-back coding.
method Introduces autoregressive sub-pixel convolution and autoregressive initial bits.
result Achieves state-of-the-art compression performance with fewer model parameters.
GeneCAI optimizes DNN compression hyper-parameters for mobile devices.
problem Efficient deployment of complex DNNs on resource-limited devices.
method GeneCAI uses genetic algorithm to learn optimal hyper-parameters.
result GeneCAI finds models with better accuracy-complexity trade-off.
Data-Dependent Coresets for Compressing Neural Networks with Applications to Generalization Boundscs.LG
We present an efficient coresets-based neural network compression algorithm that sparsifies the parameters of a trained fully-connected neural network in a manner that provably approximates the network's output. Our approach is based on an importance sampling scheme that judiciously defines a sampling distribution over…
Kernel Quantization improves CNN compression without sacrificing performance.
problem Efficiently compressing CNN models without significant performance loss.
method Quantizes convolution kernels as the unit, learning a codebook for low-bit indexes.
result Significant compression ratio achieved with minimal accuracy loss.
Differentiable model compression adds noise to parameters during training.
problem Model compression for efficient deployment.
method Adding pseudo quantization noise to model parameters during training.
result Compresses model by more than 8x on ImageNet with 0.3% accuracy loss.
Auto-Compressing Subset Pruning reduces model size for faster inference.
problem High parameter counts and slow inference times in semantic segmentation models.
method Learning a channel selection mechanism based on temperature annealing schedule.
result Significant compression of segmentation models with acceptable inference performance.
Generative Adversarial Networks (GANs) have been used in several machine learning tasks such as domain transfer, super resolution, and synthetic data generation. State-of-the-art GANs often use tens of millions of parameters, making them expensive to deploy for applications in low SWAP (size, weight, and power) hardwar…
The enormous size of modern deep neural networks makes it challenging to deploy those models in memory and communication limited scenarios. Thus, compressing a trained model without a significant loss in performance has become an increasingly important task. Tremendous advances has been made recently, where the main te…
We describe a simple and general neural network weight compression approach, in which the network parameters (weights and biases) are represented in a "latent" space, amounting to a reparameterization. This space is equipped with a learned probability model, which is used to impose an entropy penalty on the parameter r…
New theory controls compression change probability without prior knowledge.
problem Controlling compression change probability without prior knowledge.
method New theory on compression function and statistical risk.
result Cardinality of compressed set is a consistent estimator of probability of change of compression.
This work compresses heavy-tailed weight matrices for tighter generalization bounds.
problem Empirical evidence linking heavy-tailed weight matrices to test set accuracy but lack of formal relationship with generalization bounds.
method Utilized the compression framework to show that heavy-tailed matrices can be compressed, resulting in sparse weight matrices.
result Demonstrated a non-vacuous generalization bound for compressed networks with heavy-tailed weight matrices.
Paper proposes a compression principle for neural networks using Bayesian optimization.
problem Finding methods for making generalizable predictions in machine learning.
method Compression principle and Bayesian optimization approach.
result Optimal predictive models minimize total compressed message length of data and model definition.
Deep neural networks have become commonplace in the domain of reinforcement learning, but are often expensive in terms of the number of parameters needed. While compressing deep neural networks has of late assumed great importance to overcome this drawback, little work has been done to address this problem in the conte…
Dirichlet pruning compresses neural networks by removing unimportant units.
problem Compressing large neural network models without sacrificing performance.
method Assigns Dirichlet distribution over network layers' units and uses variational inference to estimate parameters.
result Achieves state-of-the-art compression performance on larger architectures like VGG and ResNet.
Adjoined Networks trains both base and compressed networks together for efficient model compression.
problem Efficiently compressing deep neural networks while maintaining accuracy.
method Adjoined Networks (AN) trains both a base network and a smaller compressed network simultaneously, sharing parameters.
result AN achieves 71.8% top-1 accuracy with 1.8M parameters and 1.6 GFLOPs on ImageNet.
On Information Geometry and Iterative Optimization in Model Compression: Operator Factorizationcs.LG
The paper uses information geometry to analyze model compression techniques, focusing on operator factorization.
problem Efficiently compressing deep learning models for resource-constrained devices.
method Information geometry applied to model compression, focusing on operator factorization.
result Iterative methods are crucial for fine-tuning models, especially when compression ratios are fixed.
We study the use of knowledge distillation to compress the U-net architecture. We show that, while standard distillation is not sufficient to reliably train a compressed U-net, introducing other regularization methods, such as batch normalization and class re-weighting, in knowledge distillation significantly improves …
Paper uses neural networks to compress large portfolios of options, reducing risk and capital requirements.
problem Managing risk and capital requirements for large portfolios of financial options.
method Artificial neural network framework for portfolio compression, static hedging, and risk management.
result The compressed portfolio's risk profiles align closely with the target portfolio's, reducing capital requirements.
This paper compresses neural networks by permuting and quantizing weights.
problem Efficiently compressing large neural networks for resource-constrained platforms.
method Permuting and quantizing weights, connecting to rate-distortion theory, and using annealed quantization.
result Significant compression with minimal accuracy loss, e.g., 40-70% reduction in gap with uncompressed model.
Recurrent Neural Networks (RNN) can be difficult to deploy on resource constrained devices due to their size.As a result, there is a need for compression techniques that can significantly compress RNNs without negatively impacting task accuracy. This paper introduces a method to compress RNNs for resource constrained e…
Compressing neural nets is an active research problem, given the large size of state-of-the-art nets for tasks such as object recognition, and the computational limits imposed by mobile devices. We give a general formulation of model compression as constrained optimization. This includes many types of compression: quan…
Proposes a new linearity-based neural network compression method.
problem Reduction of neural network model size while maintaining accuracy.
method Integrates linearity-based intuition with ReLU activation functions to merge layers.
result Achieves up to 75% reduction in model size without loss of accuracy.
New method reduces PDE model parameters by 30% with sparsity.
problem Redundant parameters in neural network projections.
method Bregman iterations for sparsity, POD compression, bias propagation.
result 30% fewer parameters with similar accuracy.
In natural language processing, a lot of the tasks are successfully solved with recurrent neural networks, but such models have a huge number of parameters. The majority of these parameters are often concentrated in the embedding layer, which size grows proportionally to the vocabulary length. We propose a Bayesian spa…
Paper introduces a new adaptive gradient method with gradient compression for distributed training.
problem Communication overhead in distributed machine learning systems.
method Adaptive gradient method with gradient compression, scalable system BytePS-Compress.
result Convergence rate of for non-convex problems.
Robust Bayesian compressive sensing with data loss recovery for structural health monitoring signalsstat.AP
The application of compressive sensing (CS) to structural health monitoring is an emerging research topic. The basic idea in CS is to use a specially-designed wireless sensor to sample signals that are sparse in some basis (e.g. wavelet basis) directly in a compressed form, and then to reconstruct (decompress) these si…
In this paper, we derive Hybrid, Bayesian and Marginalized Cramér-Rao lower bounds (HCRB, BCRB and MCRB) for the single and multiple measurement vector Sparse Bayesian Learning (SBL) problem of estimating compressible vectors and their prior distribution parameters. We assume the unknown vector to be drawn from a compr…
CoDeQ simplifies joint model compression by integrating pruning and quantization.
problem Joint pruning and quantization methods are complex and require additional procedures.
method CoDeQ uses a dead-zone quantizer to directly induce sparsity and learn quantization parameters.
result CoDeQ achieves high sparsity and low-precision accuracy with minimal bit operations.
A field known as Compressive Sensing (CS) has recently emerged to help address the growing challenges of capturing and processing high-dimensional signals and data sets. CS exploits the surprising fact that the information contained in a sparse signal can be preserved in a small number of compressive (or random) linear…
New coherence parameter for GNNs with Fourier measurements improves signal recovery.
problem Characterizing generative compressed sensing with Fourier measurements.
method Subspace counting arguments and high-dimensional probability theory.
result First known restricted isometry guarantee for generative compressed sensing with subsampled isometries.
Flexible framework compresses models using LC algorithm.
problem Efficiently compressing neural networks for resource constraints.
method Decouples learning and compression steps with alternating L and C phases.
result Compressed models maintain performance and accuracy.
Paper compresses deep neural networks by eliminating redundant neurons.
problem Challenges in deploying deep learning models due to high parameter count.
method Exploits non-linear redundancy to compress neural networks without loss.
result Reduces network size by up to 99% with minimal performance loss.
Highly distributed training of Deep Neural Networks (DNNs) on future compute platforms (offering 100 of TeraOps/s of computational capacity) is expected to be severely communication constrained. To overcome this limitation, new gradient compression techniques are needed that are computationally friendly, applicable to …
SPARQ-SGD: Event-Triggered and Compressed Communication in Decentralized Stochastic Optimizationstat.ML
In this paper, we propose and analyze SPARQ-SGD, which is an event-triggered and compressed algorithm for decentralized training of large-scale machine learning models. Each node can locally compute a condition (event) which triggers a communication where quantized and sparsified local model parameters are sent. In SPA…
Paper compresses ViTs models by 60% with minimal accuracy loss.
problem Memory constraints in ViTs models.
method Activation-aware low-rank tensor approximations.
result 60% reduction in model size with <1% accuracy loss.
Compressed sensing in MRI enables high subsampling factors while maintaining diagnostic image quality. This technique enables shortened scan durations and/or improved image resolution. Further, compressed sensing can increase the diagnostic information and value from each scan performed. Overall, compressed sensing has…
Improves convergence speed in compressive sensing with a new probabilistic approach.
problem Efficiently solving the best subset selection problem in compressive sensing.
method Smooth probabilistic reformulation of regularized regression.
result Empirically outperforms existing compressive sensing algorithms across various settings.
Deep neural networks (DNNs) have been quite successful in solving many complex learning problems. However, DNNs tend to have a large number of learning parameters, leading to a large memory and computation requirement. In this paper, we propose a model compression framework for efficient training and inference of deep …
We present a new similarity measure based on information theoretic measures which is superior than Normalized Compression Distance for clustering problems and inherits the useful properties of conditional Kolmogorov complexity. We show that Normalized Compression Dictionary Size and Normalized Compression Dictionary En…
New method reduces communication in deep learning training.
problem Communication overhead in distributed deep learning training.
method Random-block sparsification to reduce gradients communicated.
result Performance close to standard SGD with reduced communication.
We give first examples of finitely generated groups having an intermediate, with values in (0,1), Hilbert space compression (which is a numerical parameter measuring the distortion required to embed a metric space into Hilbert space). These groups include certain diagram groups. In particular, we show that the Hilbert …
This study compresses BERT to make it suitable for low-capability devices.
problem Large Transformer models are resource-intensive.
method Survey and analysis of model compression techniques for BERT.
result Insights into best practices for compressing large-scale Transformer models.