New pruning technique reduces index size for DNNs.
problem Irregular index form in fine-grained pruning limits parallelism and memory usage.
method Proposes a low-rank binary index matrix for efficient compression and decompression.
result Fine-grained pruning with binary matrices achieves lower memory footprint and higher parallelism.
Paper ranks stocks by compression risk, not volatility.
problem Investment risk not correlated with stock price volatility.
method Binary-ternary compressive coding of price change time series.
result Compression risk is a better indicator of stock investment risk.
Kernel Quantization improves CNN compression without sacrificing performance.
problem Efficiently compressing CNN models without significant performance loss.
method Quantizes convolution kernels as the unit, learning a codebook for low-bit indexes.
result Significant compression ratio achieved with minimal accuracy loss.
Convolutional neural networks (CNNs) achieve state-of-the-art performance in a wide variety of tasks in computer vision. However, interpreting CNNs still remains a challenge. This is mainly due to the large number of parameters in these networks. Here, we investigate the role of compression and particularly pruning fil…
Gene expression programming predicts compression index of fine-grained soils efficiently.
problem Estimating the compression index of fine-grained soils is costly and time-consuming.
method Developed a gene expression programming model using soil parameters.
result The GEP model predicts the compression index more accurately than conventional methods.
Compression affects deep networks differently, impacting underrepresented data points.
problem Disparate impact of compression on different classes and images.
method Analysis of deep neural network pruning and quantization effects.
result Compression disproportionately impacts model performance on underrepresented data points.
GrateTile optimizes CNN feature map storage for efficient data access.
problem Efficient storage and access of sparse CNN feature maps.
method Divides feature maps into uneven-sized subtensors, compresses and stores them in a compressed yet accessible format.
result Average 55% DRAM bandwidth reduction with minimal indexing overhead.
New method approximates M-estimator and predictions without solving fixed-point equations.
problem Characterize behavior of M-estimator and predictions in single index models.
method Develops data-driven observable adjustments to proximal operators.
result Empirical distributions of M-estimator and predictions are approximated without solving fixed-point equations.
Paper introduces a new measure combining entropy and Gini index.
problem Quantifying complexity in socio- and econo-physics.
method Generalizes entropy using Gini index and Lorenz curve transformation.
result Supports quantifying complexity in socio- and econo-physics.
PCNN prunes CNN weights efficiently for hardware acceleration.
problem Efficiently compressing CNN models for hardware acceleration.
method PCNN uses a novel Sparsity Pattern Mask (SPM) to encode and prune weights.
result PCNN achieves up to 8.4X compression with minimal accuracy loss.
The current trend of pushing CNNs deeper with convolutions has created a pressing demand to achieve higher compression gains on CNNs where convolutions dominate the computation and parameter amount (e.g., GoogLeNet, ResNet and Wide ResNet). Further, the high energy consumption of convolutions limits its deployment on m…
For a knot K ⊂ S 3 K\subset S^3 K ⊂ S 3 , its exterior E ( K ) = S 3 \ η ( K ) E(K) = S^3\backslashη(K) E ( K ) = S 3 \ η ( K ) has a singular foliation by Seifert surfaces of K K K derived from a circle-valued Morse function f : E ( K ) → S 1 f\colon E(K)\to S^1 f : E ( K ) → S 1 . When f f f is self-indexing and has no critical points of index 0 or 3, the regular levels that separate the index-1 and index-2 critica…
New insights into image compression trade-offs with private randomness.
problem Trade-off between compression rate and perceptual quality in image compression.
method Characterization of rate-distortion trade-off with private randomness under different realism constraints.
result Encoder private randomness is not useful if compression rate is below source entropy, even with limited common and decoder private randomness.
Bayesian model averaging under predictor redundancy
problem Reporting Bayesian model averaging posterior without changing the Bayesian target
method Using hard or soft regions of support space
result Region reports often give shorter and clearer summaries while preserving the main posterior information
It is inevitable to train large deep learning models on a large-scale cluster equipped with accelerators system. Deep gradient compression would highly increase the bandwidth utilization and speed up the training process but hard to implement on ring structure. In this paper, we find that redundant gradient and gradien…
Study on recovering supports of multiple sparse vectors from mixed linear measurements.
problem Recovering supports of multiple sparse vectors from a mixture of linear measurements.
method Developed algorithms to identify the support of all component vectors using polynomial and quasi-polynomial number of measurements.
result Polynomial and quasi-polynomial number of measurements sufficient for recovering the supports of all component vectors.
This paper improves support recovery in universal one-bit compressed sensing with fewer measurements.
problem Support recovery in universal one-bit compressed sensing.
method Developed algorithms to recover the support of sparse signals with a small number of false positives.
result Support recovery with i l d e O ( k 3 / 2 ) ilde{O}(k^{3/2}) i l d e O ( k 3/2 ) measurements, improving to i l d e O ( k ) ilde{O}(k) i l d e O ( k ) with known dynamic range. We perform wavelet decomposition of high frequency financial time series into large and small time scale components. Taking the FTSE100 index as a case study, and working with the Haar basis, it turns out that the small scale component defined by most ( ≃ \simeq ≃ 99.6%) of the wavelet coefficients can be neglected for th…
Survival kernets scale deep kernel survival analysis to large datasets with interpretability and theoretical guarantees.
problem Scalable and interpretable deep kernel survival analysis for large datasets.
method Survival kernets use kernel netting for training set compression and XGBoost for warm-starting neural architecture search.
result Survival kernets achieve optimal time-dependent concordance index on various survival analysis datasets.
Randomness and regularities in Finance are usually treated in probabilistic terms. In this paper, we develop a completely different approach in using a non-probabilistic framework based on the algorithmic information theory initially developed by Kolmogorov (1965). We present some elements of this theory and show why i…
Unified method learns latent spaces from labeled data.
problem Identifying low-dimensional latent structures in high-dimensional data.
method Nonlinear multiple-response regression within an index model context.
result Unified method offers better interpretability and reduced computational complexity.
Unified framework for uniform signal recovery in nonlinear GCS with 1-bit/quantized measurements.
problem Uniform recovery guarantees for nonlinear generative compressed sensing.
method Unified framework using generalized Lasso and Lipschitz approximation.
result Uniform recovery of all signals in the ball up to an error of ε using approximately O(k/ε^2) samples.
Flexible framework compresses models using LC algorithm.
problem Efficiently compressing neural networks for resource constraints.
method Decouples learning and compression steps with alternating L and C phases.
result Compressed models maintain performance and accuracy.
Reduces multiclass and regression compression schemes to binary ones.
problem Developing efficient learning algorithms for multiclass and regression problems.
method Reduces sample compression schemes for binary classes to multiclass and regression settings.
result Establishes new compression schemes for multiclass and regression problems.
Proposes a link between randomness and compression in deep learning.
problem Improving efficiency in deep learning training.
method Introduces a novel tomographic compression framework called Dual Tomographic Compression (DTC).
result Demonstrates high correlation between learning performance and Gibbs entropy over compression ratios.
Unified framework for non-compressed networks from compression bounds.
problem Generalization of large deep neural networks with huge model size.
method Unified framework converting compression based bounds to non-compressed networks.
result Unified data-dependent generalization error bound with tighter evaluation.
Paper introduces a new adaptive gradient method with gradient compression for distributed training.
problem Communication overhead in distributed machine learning systems.
method Adaptive gradient method with gradient compression, scalable system BytePS-Compress.
result Convergence rate of O ( 1 / T ) \mathcal{O}(1/\sqrt{T}) O ( 1/ T ) for non-convex problems. Galen algorithm compresses neural networks for specific hardware with reduced latency.
problem Finding optimal compression policies for neural networks on specific hardware.
method Reinforcement learning using pruning and quantization to optimize inference latency.
result Compressed ResNet18 for ARM processor reduced inference latency by 80%.
The (isothermic) compressibility of lattice knots can be examined as a model of the effects of topology and geometry on the compressibility of ring polymers. In this paper, the compressibility of minimal length lattice knots in the simple cubic, face centered cubic and body centered cubic lattices are determined. Our r…
Paper compresses neural network weight-updates for image artifacts removal.
problem Efficiently compressing neural network weight-updates for image artifacts removal.
method Fine-tuning a pre-trained artifact removal network on target data with a compression objective that encourages sparse and quantized weight-updates.
result Achieves reconstruction quality comparable to traditional codecs at comparable bitrates.
HiLLoC compresses large images losslessly using VAEs.
problem Lossless compression of large color photographs.
method Fully convolutional VAE models trained on ImageNet are applied to lossless compression.
result Achieves state-of-the-art compression for full-size ImageNet images.
Neural NCD reveals LLMs don't compress well for classification.
problem The disconnect between compression and classification in neural networks.
method Developed Neural NCD to compare LLMs to classic algorithms, finding classification accuracy not correlated with compression rate.
result Classification accuracy is not predictable by compression rate alone, challenging current understanding.
BDC compresses both sample size and dimensionality of large datasets.
problem Large datasets in both sample size and dimensionality.
method Two-stage framework using Decoded MMD, Reconstruction MMD, and Encoded MMD.
result BDC achieves comparable or superior performance with lower cost and higher compression rates.
This paper compresses RNNs for IoT devices by 15-38x using Kronecker products.
problem Resource constraints on IoT devices make RNNs difficult to deploy.
method Kronecker product (KP) for compressing RNN layers by 15-38x with minimal accuracy loss.
result Kronecker product can compress RNNs by 50x when quantized to 8-bits.
DoCoFL compresses model updates for cross-device federated learning.
problem Downlink compression for cross-device federated learning where clients may appear only once.
method Proposes DoCoFL framework for downlink compression in cross-device federated learning.
result Significant bi-directional bandwidth reduction with competitive accuracy.
Compressing neural nets is an active research problem, given the large size of state-of-the-art nets for tasks such as object recognition, and the computational limits imposed by mobile devices. We give a general formulation of model compression as constrained optimization. This includes many types of compression: quan…
Paper presents a new video compression method using autoencoders.
problem Efficient video compression with reduced quality loss.
method 3D autoencoder with discrete latent space and autoregressive prior trained jointly.
result Method outperforms state-of-the-art learned video compression networks.
New taxonomy and evaluation of neural network compression methods.
problem Efficiency of deep neural networks in real-world applications.
method Categorization and evaluation of tensor factorization and probabilistic compression methods.
result SVD and probabilistic compression methods are complementary and give the best results.
Proposes a method to train neural networks directly on compressed text data.
problem Training neural networks on compressed text data without decompression.
method Introduces composer modules to encode symbols from grammar compression rules into vector representations.
result Demonstrates that the proposed method can achieve both memory and computational efficiency while maintaining moderate performance.
Study shows LLC correlates with neural network compressibility.
problem Evaluating limits of neural network compression.
method Extended minimum description length principle using singular learning theory.
result Complexity estimates based on LLC are linearly correlated with compressibility.
Paper proposes SCALLION and SCAFCOM for compressed FL with reduced communication.
problem Reducing communication overhead in Federated Learning with data heterogeneity and partial participation.
method Revisit and simplify stochastic controlled averaging, proposing SCALLION and SCAFCOM for unbiased and biased compression.
result SCALLION and SCAFCOM outperform existing methods in communication and computation complexities.
Gradient descent with compressed iterates reduces model size for federated learning.
problem Efficiently compressing model sizes in federated learning.
method Gradient descent with iterates compressed using lossy randomized techniques.
result Improves model compression and aggregation in federated learning.
New uncertainty principle limits compression in distributed learning, suggesting optimal methods.
problem Minimizing communication cost while maintaining message quality in distributed learning.
method Formalized uncertainty principle for compression operators, introduced Kashin compression.
result Explicit formula for Kashin compression's variance bound, dimension independent.
Proposes a new linearity-based neural network compression method.
problem Reduction of neural network model size while maintaining accuracy.
method Integrates linearity-based intuition with ReLU activation functions to merge layers.
result Achieves up to 75% reduction in model size without loss of accuracy.
Paper introduces compressibility loss for learning sparse neural network weights.
problem Learning highly compressible neural network weights.
method Applying a compressibility loss to minimize the negated sparsity of the signal.
result At critical points, weight vectors are ternary signals with a sparsity directly related to the objective value.
Paper compresses RNNs for resource-constrained devices.
problem Difficulty deploying RNNs on resource-constrained devices.
method Uses Kronecker product (KP) to compress RNN layers.
result KP compresses RNN layers by 16-38x with minimal accuracy loss.
Randomized matrix compression techniques, such as the Johnson-Lindenstrauss transform, have emerged as an effective and practical way for solving large-scale problems efficiently. With a focus on computational efficiency, however, forsaking solutions quality and accuracy becomes the trade-off. In this paper, we investi…
Can textual data be compressed intelligently without losing accuracy in evaluating sentiment? In this study, we propose a novel evolutionary compression algorithm, PARSEC (PARts-of-Speech for sEntiment Compression), which makes use of Parts-of-Speech tags to compress text in a way that sacrifices minimal classification…