To improve how neural networks function it is crucial to understand their learning process. The information bottleneck theory of deep learning proposes that neural networks achieve good generalization by compressing their representations to disregard information that is not relevant to the task. However, empirical evid…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,694 papers · 148 categories
Trend · papers per month
48 results for “Activation Compression”
IMPACT optimizes LLM compression by focusing on activation importance, reducing model size up to 55.4%.
problem Resource constraints in deploying large language models (LLMs).
method IMPACT integrates activation importance into low-rank compression, optimizing for both size and accuracy.
result IMPACT achieves up to 55.4% greater model size reduction while maintaining comparable or better accuracy.
Proposes a new linearity-based neural network compression method.
problem Reduction of neural network model size while maintaining accuracy.
method Integrates linearity-based intuition with ReLU activation functions to merge layers.
result Achieves up to 75% reduction in model size without loss of accuracy.
Cross-Layer Subspace Coupling for LLM Compression: A Unifying Framework and Its Empirical Limitscs.LG
Unified SVD compression fails in practical tasks, highlighting the importance of per layer activation reconstruction.
problem The failure of a unified SVD compression method in practical tasks like perplexity and accuracy.
method Unified optimization problem for SVD based compression methods, focusing on cross-layer coupling.
result Downstream metrics like perplexity and accuracy degrade severely compared to standard per layer SVD LLM.
LASER compresses recursive model activations by exploiting their low-dimensional structure.
problem Understanding and optimizing the geometric structure of recursive reasoning trajectories.
method Dynamic low-rank basis tracking via matrix-free subspace tracking with a fidelity-triggered reset mechanism.
result Recursive activations occupy a linear, low-dimensional subspace that can be compressed efficiently.
We introduce a new and improved characterization of the label complexity of disagreement-based active learning, in which the leading quantity is the version space compression set size. This quantity is defined as the size of the smallest subset of the training data that induces the same version space. We show various a…
We consider the problems of detection and localization of a contiguous block of weak activation in a large matrix, from a small number of noisy, possibly adaptive, compressive (linear) measurements. This is closely related to the problem of compressed sensing, where the task is to estimate a sparse vector using a small…
Improved EXACT strategy reduces GNN memory consumption and runtime.
problem Efficiently training large-scale GNNs with reduced memory usage.
method Block-wise quantization of intermediate activation maps with improved variance minimization.
result Further reduction in memory consumption (>15%) and runtime speedup (5%) with similar performance trade-offs.
Paper compresses ViTs models by 60% with minimal accuracy loss.
problem Memory constraints in ViTs models.
method Activation-aware low-rank tensor approximations.
result 60% reduction in model size with <1% accuracy loss.
Paper proposes ADC framework to reduce ViT SL training communication overhead.
problem Reducing communication overhead in ViT SL training.
method Two parallel compression strategies: class-agnostic merging and token discarding.
result Significantly reduces communication overhead without sacrificing accuracy.
We study the localization of a cluster of activated vertices in a graph, from adaptively designed compressive measurements. We propose a hierarchical partitioning of the graph that groups the activated vertices into few partitions, so that a top-down sensing procedure can identify these partitions, and hence the activa…
ActNN reduces neural network training memory by 2-bit quantization.
problem Limited memory in training neural networks.
method Randomly quantizes activations to 2 bits, proving convergence and proposing mixed-precision strategies.
result Reduces activation memory footprint by 12x with negligible accuracy loss.
This paper introduces a new measure to identify model redundancy in compressed CNNs.
problem Identifying remaining model redundancy in compressed CNNs.
method Developed a statistical formulation of CNNs and compressed CNNs via tensor decomposition, revealing discrepancies in sample complexity and model redundancy.
result Introduced a new model redundancy measure, the ratio, for compressed CNNs.
Study reduces memory needs for active learning with enriched queries.
problem Expensive labeling costs in active learning.
method Introduces bounded memory active learning through enriched queries, introduces lossless sample compression.
result Can learn classifiers with bounded memory and query optimality.
We introduce a conceptually simple and scalable framework for continual learning domains where tasks are learned sequentially. Our method is constant in the number of parameters and is designed to preserve performance on previously encountered tasks while accelerating learning progress on subsequent problems. This is a…
New algorithms allow multiple robots to search efficiently without central coordination.
problem Efficiently locating targets in an unknown environment with multiple robots.
method SPATS and LATSI algorithms for asynchronous multi-agent active search.
result Multiple robots can independently make data-collection decisions without a central coordinator.
Paper compresses deep neural networks by eliminating redundant neurons.
problem Challenges in deploying deep learning models due to high parameter count.
method Exploits non-linear redundancy to compress neural networks without loss.
result Reduces network size by up to 99% with minimal performance loss.
New method corrects quantization errors in LLMs using low-rank matrices.
problem Correcting quantization errors in large language models.
method Introducing low-rank weight matrices to correct quantized activations in LLMs.
result Reduces accuracy gap with original model by more than 50% using low-rank matrices.
A fast method for Lasso and Logistic Lasso problems.
problem Solving Lasso and Logistic Lasso regression problems efficiently.
method Iterative active set approach using solver updates.
result 31.41 times faster on average for compressed sensing.
New method preserves privacy while improving machine learning accuracy.
problem Privacy-preserving machine learning for daily data.
method Compressive Privacy and multi-kernel method.
result Improved utility classification accuracy with privacy preservation.
With the rapid scaling up of deep neural networks (DNNs), extensive research studies on network model compression such as weight pruning have been performed for improving deployment efficiency. This work aims to advance the compression beyond the weights to neuron activations. We propose the joint regularization techni…
Recent literature on unsupervised learning focused on designing structural priors with the aim of learning meaningful features, but without considering the description length of the representations. In this thesis, first we introduce the metric that evaluates unsupervised models based on their reconstruction …
Compressing neural nets is an active research problem, given the large size of state-of-the-art nets for tasks such as object recognition, and the computational limits imposed by mobile devices. We give a general formulation of model compression as constrained optimization. This includes many types of compression: quan…
Method scales up ML science by measuring multiple molecules at once.
problem Scaling up ML-driven science with wet lab experiments.
method Neural extension of compressed sensing for function space.
result Proves orders-of-magnitude gains in information density.
To improve the execution speed and efficiency of neural networks in embedded systems, it is crucial to decrease the model size and computational complexity. In addition to conventional compression techniques, e.g., weight pruning and quantization, removing unimportant activations can reduce the amount of data communica…
In this manuscript we propose two objective terms for neural image compression: a compression objective and a cycle loss. These terms are applied on the encoder output of an autoencoder and are used in combination with reconstruction losses. The compression objective encourages sparsity and low entropy in the activatio…
We consider the problem of compressed sensing and of (real-valued) phase retrieval with random measurement matrix. We derive sharp asymptotics for the information-theoretically optimal performance and for the best known polynomial algorithm for an ensemble of generative priors consisting of fully connected deep neural …
Hierarchical Spatio-Channel Clustering for Efficient Model Compression in Medical Image Analysiscs.CV
This paper proposes a new method to compress CNNs for medical image analysis, improving efficiency and accuracy.
problem Large memory and computational requirements of CNNs in resource-constrained environments.
method Hierarchical spatio-channel low-rank compression framework that partitions feature maps into spatial regions and groups channels according to co-activation patterns within each region.
result The proposed method achieves significant FLOP reduction, inference speed-up, and improved classification accuracy compared to existing methods.
TinyLSTMs reduces speech enhancement model size and latency for hearing aids.
problem Large RNNs limit practical deployment in hearing aid hardware.
method Model compression techniques (pruning, integer quantization, state update skipping) for RNN speech enhancement.
result Reduction in model size and operations by 11.9 and 2.9, respectively, without perceptual degradation.
Neural network compression methods have enabled deploying large models on emerging edge devices with little cost, by adapting already-trained models to the constraints of these devices. The rapid development of AI-capable edge devices with limited computation and storage requires streamlined methodologies that can effi…
Over-the-counter markets are at the center of the postcrisis global reform of the financial system. We show how the size and structure of such markets can undergo rapid and extensive changes when participants engage in portfolio compression, a post-trade netting technology. Tightly-knit and concentrated trading structu…
Deep Neural Networks (DNNs) have recently been achieving state-of-the-art performance on a variety of computer vision related tasks. However, their computational cost limits their ability to be implemented in embedded systems with restricted resources or strict latency constraints. Model compression has therefore been …
ResNet learns to compress information during training.
problem Applying information bottleneck principle to deep learning models.
method Trained PixelCNN++ models to measure mutual information between ResNet hidden layers and input data.
result Compression occurs in both classification and autoencoding training regimes.
The measurement and analysis of Electrodermal Activity (EDA) offers applications in diverse areas ranging from market research, to seizure detection, to human stress analysis. Unfortunately, the analysis of EDA signals is made difficult by the superposition of numerous components which can obscure the signal informatio…
"Lossless" Compression of Deep Neural Networks: A High-dimensional Neural Tangent Kernel Approachstat.ML
Lossless compression of deep neural networks using NTK and RMT.
problem Compressing large-scale deep neural networks for low-power devices.
method High-dimensional neural tangent kernel approach.
result Asymptotic spectral equivalence between NTK matrices of wide DNNs enables lossless compression.
Clapping reduces memory usage in distributed optimization by reusing data samples.
problem Significant communication overhead and impractical memory overhead in pipeline-parallel distributed optimization.
method Lazy sampling strategy to reuse data samples across steps, supporting convergence without unbiased gradient assumptions.
result Clapping achieves convergence in few-epoch or online training regimes without sample-size memory overhead.
This paper analyzes the training dynamics of binary neural networks using information bottleneck.
problem Training binary neural networks is challenging due to discontinuity in activation functions.
method The approach uses the Information Bottleneck principle to analyze BNN training dynamics.
result Training dynamics of BNNs are different from DNNs, with both phases occurring simultaneously.
Highly distributed training of Deep Neural Networks (DNNs) on future compute platforms (offering 100 of TeraOps/s of computational capacity) is expected to be severely communication constrained. To overcome this limitation, new gradient compression techniques are needed that are computationally friendly, applicable to …
This paper studies how to compress neural networks while maintaining accuracy.
problem Compressing a two-layer neural network with fewer nodes without losing accuracy.
method Using tools from high-dimensional probability, the authors minimize the L_2 loss between the target and compressed networks.
result The error rate of the approximation is shown as a function of input dimension and network size in the mean-field limit.
The paper explores tensor decompositions in deep learning models.
problem Compressing parameter space and creating richer representations.
method Tensor decompositions applied to deep learning models.
result Tensor methods can yield richer adaptive representations of complex data.
A Primal Dual Active Set with Continuation Algorithm for the \ell^0-Regularized Optimization Problemmath.OC
We develop a primal dual active set with continuation algorithm for solving the \ell^0-regularized least-squares problem that frequently arises in compressed sensing. The algorithm couples the the primal dual active set method with a continuation strategy on the regularization parameter. At each inner iteration, it fir…
Neural network pruning with suitable retraining can yield networks with considerably fewer parameters than the original with comparable degrees of accuracy. Typical pruning methods require large, fully trained networks as a starting point from which they perform a time-intensive iterative pruning and retraining procedu…
Deep neural networks are often used to implement powerful generative models for real-world data. Notable applications include image denoising, as well as other classical inverse problems like compressed sensing and super-resolution. To provide a rigorous but simplified analysis of generative models, in this work, we in…
Large Learning Rates Simultaneously Achieve Robustness to Spurious Correlations and Compressibilitycs.LG
Large learning rates enhance model robustness and compressibility.
problem Achieving robustness and resource-efficiency in machine learning models.
method Identifying and utilizing large learning rates as a facilitator for robustness and compressibility.
result Large learning rates produce desirable representation properties and compare favorably to other methods.
Explaining the prediction of deep neural networks (DNNs) and semantic image compression are two active research areas of deep learning with a numerous of applications in decision-critical systems, such as surveillance cameras, drones and self-driving cars, where interpretable decision is critical and storage/network ba…
Efficient algorithms for monophonic halfspaces in graphs simplify learning and compression.
problem Learning and compressing monophonic halfspaces in graphs.
method 2-satisfiability based decomposition theorem, efficient algorithms for various learning problems.
result Achieved efficient and nearly optimal algorithms for various learning problems.
Previous literature on unsupervised learning focused on designing structural priors with the aim of learning meaningful features. However, this was done without considering the description length of the learned representations which is a direct and unbiased measure of the model complexity. In this paper, first we intro…
Improves active learning efficiency by warping input space based on observed outputs.
problem Insensitivity of Gaussian process uncertainty to actual observations.
method Input warping with learned monotone reparameterization to adjust acquisition function behavior.
result Significantly improved sample efficiency across various benchmarks, especially in non-stationary conditions.