arXiv research

New coding theorem shows achievable rate matches theoretical limit.

Estimates rate-distortion function for large datasets using neural networks.

Lossy compression algorithms are typically designed and analyzed through the lens of Shannon's rate-distortion theory, where the goal is to achieve the lowest possible distortion (e.g., low MSE or high SSIM) at any given bit rate. However, in recent years, it has become increasingly accepted that "low distortion" is no…

Predictive rate-distortion analysis suffers from the curse of dimensionality: clustering arbitrarily long pasts to retain information about arbitrarily long futures requires resources that typically grow exponentially with length. The challenge is compounded for infinite-order Markov processes, since conditioning on fi…

Paper studies fundamental limits of communication in distributed learning.

New method optimizes lossy compression models more effectively.

Optimizes communication in federated learning using rate-distortion theory.

We propose computationally efficient encoders and decoders for lossy compression using a Sparse Regression Code. The codebook is defined by a design matrix and codewords are structured linear combinations of columns of this matrix. The proposed encoding algorithm sequentially chooses columns of the design matrix to suc…

New bounds for optimal transport using Gaussian processes and rate-distortion functions.

Modified BA algorithm computes RD and DR functions efficiently.

High-dimensional VAEs inevitably collapse to prior, requiring large datasets for good performance.

New method estimates rate-distortion function using optimal transport.

Researchers prove inner product recovery is impossible in latent space models.

This paper explores the problem of learning transforms for image compression via autoencoders. Usually, the rate-distortion performances of image compression are tuned by varying the quantization step size. In the case of autoen-coders, this in principle would require learning one transform per rate-distortion point at…

New method improves bivariate causal discovery by accurately estimating cause variable complexity.

New bounds on generalization error for distributed learning using rate-distortion theory.

This paper studies communication efficiency in federated learning by optimizing the sum-rate-distortion function for indirect multiterminal source coding.

In this paper, the `Approximate Message Passing' (AMP) algorithm, initially developed for compressed sensing of signals under i.i.d. Gaussian measurement matrices, has been extended to a multi-terminal setting (MAMP algorithm). It has been shown that similar to its single terminal counterpart, the behavior of MAMP algo…

New compression theory justifies model pruning for neural networks.

This work adapts RDT for mental program construction, showing benefits and costs.

New approach connects robust learning to privacy issues.

Simplifies VAE for anomaly detection using rate-distortion theory.

Single training run learns optimal VAE parameters for various β values.

Lower bounds on Bayes risk for realizable models derived using information theory.

The enormous size of modern deep neural networks makes it challenging to deploy those models in memory and communication limited scenarios. Thus, compressing a trained model without a significant loss in performance has become an increasingly important task. Tremendous advances has been made recently, where the main te…

Unified bounds linking compressibility, fractal dimensions, and mutual information.

In this article we use rate-distortion theory, a branch of information theory devoted to the problem of lossy compression, to shed light on an important problem in latent variable modeling of data: is there room to improve the model? One way to address this question is to find an upper bound on the probability (equival…

Paper proposes a new black-box attack approach to minimize visual distortion.

Optimizes portfolio growth rate for a behavioral investor considering terminal relative growth rate.

New insights into optimizing latent representations in hierarchical VAEs.

We present an information-theoretic framework for bounding the number of labeled samples needed to train a classifier in a parametric Bayesian setting. We derive bounds on the average $L_p$ distance between the learned classifier and the true maximum a posteriori classifier, which are well-established surrogates for th…

This study analyzes communication constraints in MoE architectures using information theory.

This paper develops a method to estimate the rate-distortion function for general data sources.

Paper proposes a CNN-based method for estimating intra frame bits and quality.

Autoencoder optimizes data embedding for accurate PDF reproduction.

Foundation models fail to preserve continuous geometry, identified as the Geometric Alignment Tax.

New insights into image compression trade-offs with private randomness.

The paper integrates behavioral distortions into portfolio optimization using implied probability weighting functions.

VAEs improve generalization by regulating mutual information, contrary to intuition.

Study shows how high-budget agents can manipulate prediction markets.

Investigates stability of Epstein-Zin problem under market distortions.

Compression is at the heart of effective representation learning. However, lossy compression is typically achieved through simple parametric models like Gaussian noise to preserve analytic tractability, and the limitations this imposes on learning are largely unexplored. Further, the Gaussian prior assumptions in model…

Proposes flexible auto-encoders for varying data dimensions.

A distinctive property of human and animal intelligence is the ability to form abstractions by neglecting irrelevant information which allows to separate structure from noise. From an information theoretic point of view abstractions are desirable because they allow for very efficient information processing. In artifici…

A new approach for efficient data compression in split DNN computing.

In this manuscript we propose two objective terms for neural image compression: a compression objective and a cycle loss. These terms are applied on the encoder output of an autoencoder and are used in combination with reconstruction losses. The compression objective encourages sparsity and low entropy in the activatio…

Extends likelihood ratio exponential families to analyze various optimization methods.

In this paper, we continue our previous work on the Dirichlet mixture model (DMM)-based VQ to derive the performance bound of the LSF VQ. The LSF parameters are transformed into the $Δ$LSF domain and the underlying distribution of the $Δ$LSF parameters are modelled by a DMM with finite number of mixture components. The…