A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
LLMs compress financial texts, but distort decision-making.
New compression theory justifies model pruning for neural networks.
The paper uses information geometry to analyze model compression techniques, focusing on operator factorization.
ResNet learns to compress information during training.
To improve how neural networks function it is crucial to understand their learning process. The information bottleneck theory of deep learning proposes that neural networks achieve good generalization by compressing their representations to disregard information that is not relevant to the task. However, empirical evid…
Unified bounds linking compressibility, fractal dimensions, and mutual information.
Neural NCD reveals LLMs don't compress well for classification.
Asynchronous computation and gradient compression have emerged as two key techniques for achieving scalability in distributed optimization for large-scale machine learning. This paper presents a unified analysis framework for distributed gradient methods operating with staled and compressed gradients. Non-asymptotic bo…
This work compresses sequences by treating them as continuous-time processes, enabling efficient discretization.
We show that model compression can improve the population risk of a pre-trained model, by studying the tradeoff between the decrease in the generalization error and the increase in the empirical risk with model compression. We first prove that model compression reduces an information-theoretic bound on the generalizati…
Proposes a new method to selectively access privileged information in reinforcement learning.
New method reduces FL communication cost by leveraging server's side information.
Recent research has studied the role of sparsity in high dimensional regression and signal reconstruction, establishing theoretical limits for recovering sparse models from sparse data. This line of work shows that -regularized least squares regression can accurately estimate a sparse linear model from nois…
GeoIB uses information geometry to control compression in deep learning models.
New method quantifies redundant information using information bottleneck.
PowerGossip compresses model differences for decentralized deep learning with low-rank linear compressors.
This work considers an estimation task in compressive sensing, where the goal is to estimate an unknown signal from compressive measurements that are corrupted by additive pre-measurement noise (interference, or clutter) as well as post-measurement noise, in the specific setting where some (perhaps limited) prior knowl…
Paper proposes SCALLION and SCAFCOM for compressed FL with reduced communication.
The Information Plane theory predicts autoencoders do not compress input information.
LanguaShrink compresses prompts using psycholinguistic principles to reduce costs.
In this paper, we investigate the degree to which the encoding of a -VAE captures label information across multiple architectures on Binary Static MNIST and Omniglot. Even though they are trained in a completely unsupervised manner, we demonstrate that a -VAE can retain a large amount of label information, even w…
New theory controls compression change probability without prior knowledge.
Compressed sensing techniques enable efficient acquisition and recovery of sparse, high-dimensional data signals via low-dimensional projections. In this work, we propose Uncertainty Autoencoders, a learning framework for unsupervised representation learning inspired by compressed sensing. We treat the low-dimensional …
Study on reducing forgetting in neural networks using compression theory.
We consider the problem of compressed sensing and of (real-valued) phase retrieval with random measurement matrix. We derive sharp asymptotics for the information-theoretically optimal performance and for the best known polynomial algorithm for an ensemble of generative priors consisting of fully connected deep neural …
Machine learning models are generally vulnerable to adversarial examples, which is in contrast to the robustness of humans. In this paper, we try to leverage one of the mechanisms in human recognition and propose a bio-inspired classification framework in which model inference is conditioned on label hypothesis. We pro…
We study the flow of information and the evolution of internal representations during deep neural network (DNN) training, aiming to demystify the compression aspect of the information bottleneck theory. The theory suggests that DNN training comprises a rapid fitting phase followed by a slower compression phase, in whic…
Proposes Decodable Information Bottleneck for optimal representation learning.
Paper tackles noisy labels by compressing feature representations.
We describe a general framework -- compressive statistical learning -- for resource-efficient large-scale learning: the training collection is compressed in one pass into a low-dimensional sketch (a vector of random empirical generalized moments) that captures the information relevant to the considered learning task. A…
AIB method improves robustness against adversarial perturbations.
Certain biological neurons demonstrate a remarkable capability to optimally compress the history of sensory inputs while being maximally informative about the future. In this work, we investigate if the same can be said of artificial neurons in recurrent neural networks (RNNs) trained with maximum likelihood. Empirical…
Image compression techniques reveal network structure for shipping box optimization.
Review of information plane analyses in neural networks, highlighting mixed results and methodological challenges.
New bounds improve neural network generalization through slicing.
In this draft, which reports on work in progress, we 1) adapt the information bottleneck functional by replacing the compression term by class-conditional compression, 2) relax this functional using a variational bound related to class-conditional disentanglement, 3) consider this functional as a training objective for…
This paper investigates methods for quantifying similarity between audio signals, specifically for the task of of cover song detection. We consider an information-theoretic approach, where we compute pairwise measures of predictability between time series. We compare discrete-valued approaches operating on quantised au…
Proposes a link between randomness and compression in deep learning.
We address the problem of Compressed Sensing (CS) with side information. Namely, when reconstructing a target CS signal, we assume access to a similar signal. This additional knowledge, the side information, is integrated into CS via L1-L1 and L1-L2 minimization. We then provide lower bounds on the number of measuremen…
CIB compresses variables causally, preserving key causal interactions.
New approach for learning large Bayesian networks using feature clustering and compression.
Compressed sensing in MRI enables high subsampling factors while maintaining diagnostic image quality. This technique enables shortened scan durations and/or improved image resolution. Further, compressed sensing can increase the diagnostic information and value from each scan performed. Overall, compressed sensing has…
We introduce a simple recurrent variational auto-encoder architecture that significantly improves image modeling. The system represents the state-of-the-art in latent variable models for both the ImageNet and Omniglot datasets. We show that it naturally separates global conceptual information from lower level details, …
This paper presents a novel signal compression algorithm based on the Blaschke unwinding adaptive Fourier decomposition (AFD). The Blaschke unwinding AFD is a newly developed signal decomposition theory. It utilizes the Nevanlinna factorization and the maximal selection principle in each decomposition step, and achieve…
Single model corrects JPEG artifacts for various compression settings.
We present a new similarity measure based on information theoretic measures which is superior than Normalized Compression Distance for clustering problems and inherits the useful properties of conditional Kolmogorov complexity. We show that Normalized Compression Dictionary Size and Normalized Compression Dictionary En…
New decoder improves robustness of compressive clustering.
WoodFisher improves neural network compression efficiency and accuracy.