arXiv research

Sharp asymptotics derived for phase retrieval and compressed sensing with random generative priors.

The paper applies Information Bottleneck theory to CNNs and finds compression phase not always present.

New guarantees for asymmetric sketching in compressive learning.

Near-optimal sample complexity for phase retrieval with generative priors.

Proposes using GANs to solve phase retrieval problems.

In compressed sensing problems, $\ell_1$ minimization or Basis Pursuit was known to have the best provable phase transition performance of recoverable sparsity among polynomial-time algorithms. It is of great theoretical and practical interest to find alternative polynomial-time algorithms which perform better than $\e…

This paper introduces a novel generative encoder (GE) model for generative imaging and image processing with applications in compressed sensing and imaging, image compression, denoising, inpainting, deblurring, and super-resolution. The GE model consists of a pre-training phase and a solving phase. In the pre-training …

End-to-end meta-learned system for image compression.

Autoencoders fail to capture sparse structure in 1-bit data compression.

Deep neural networks as image priors have been recently introduced for problems such as denoising, super-resolution and inpainting with promising performance gains over hand-crafted image priors such as sparsity and low-rank. Unlike learned generative priors they do not require any training over large datasets. However…

The paper studies phase transitions in Information Bottleneck for representation learning.

A major issue in harmonic analysis is to capture the phase dependence of frequency representations, which carries important signal properties. It seems that convolutional neural networks have found a way. Over time-series and images, convolutional networks often learn a first layer of filters which are well localized i…

Spectral mixture (SM) kernels comprise a powerful class of generalized kernels for Gaussian processes (GPs) to describe complex patterns. This paper introduces model compression and time- and phase (TP) modulated dependency structures to the original (SM) kernel for improved generalization of GPs. Specifically, by adop…

Characterizing the phase transitions of convex optimizations in recovering structured signals or data is of central importance in compressed sensing, machine learning and statistics. The phase transitions of many convex optimization signal recovery methods such as $\ell_1$ minimization and nuclear norm minimization are…

We develop an explicit and tractable representation of a twist-grain-boundary phase of a smectic A liquid crystal. This allows us to calculate the interaction energy between grain boundaries and the relative contributions from the bending and compression deformations. We discuss the special stability of the 90 degree g…

Accelerated magnetic resonance (MR) scan acquisition with compressed sensing (CS) and parallel imaging is a powerful method to reduce MR imaging scan time. However, many reconstruction algorithms have high computational costs. To address this, we investigate deep residual learning networks to remove aliasing artifacts …

New approach selects sparse features without validation.

We study the flow of information and the evolution of internal representations during deep neural network (DNN) training, aiming to demystify the compression aspect of the information bottleneck theory. The theory suggests that DNN training comprises a rapid fitting phase followed by a slower compression phase, in whic…

Paper uses SGLD to recover signals from generative models, proving convergence under mild conditions.

Paper shows how to integrate quantization into neural compression models.

Review of information plane analyses in neural networks, highlighting mixed results and methodological challenges.

New method recovers signals from compressed measurements using generative networks with contractive layers.

New measure LMN explains neural network grokking.

In many compressive sensing problems today, the relationship between the measurements and the unknowns could be nonlinear. Traditional treatment of such nonlinear relationships have been to approximate the nonlinearity via a linear model and the subsequent un-modeled dynamics as noise. The ability to more accurately ch…

This paper proposes a new framework to regularize the highly ill-posed and non-linear phase retrieval problem through deep generative priors using simple gradient descent algorithm. We experimentally show effectiveness of proposed algorithm for random Gaussian measurements (practically relevant in imaging through scatt…

The training phases of Deep neural network~(DNN) consumes enormous processing time and energy. Compression techniques utilizing the sparsity of DNNs can effectively accelerate the inference phase of DNNs. However, it can be hardly used in the training phase because the training phase involves dense matrix-multiplicatio…

This paper analyzes the training dynamics of binary neural networks using information bottleneck.

FEDS distills LIC model knowledge into a lightweight student for efficient compression.

A new compression method reduces model complexity and improves performance.

Optimizes neural network training by dynamically updating Tucker decomposition ranks.

Improved survival analysis using square root Cox's models and neural networks.

SuperNet speeds up neural network ensembling by training a single DNN for various phases.

Proposes a new binary classification model inspired by fluid phase separation.

Optimal spectral initializers impact phase retrieval phase transitions.

In this paper, we study the problem of compressed sensing using binary measurement matrices and $\ell_1$-norm minimization (basis pursuit) as the recovery algorithm. We derive new upper and lower bounds on the number of measurements to achieve robust sparse recovery with binary matrices. We establish sufficient conditi…

Improved online Sinkhorn algorithm for large-scale data processing.

A new method enhances signal recovery with FDR control.

Gradual pruning reduces inference cost by pruning least important channels during training.

ReLU networks learn simple models even with many parameters, overcoming traditional wisdom.

SSVI efficiently trains sparse Bayesian neural networks with minimal compression and performance loss.

Neural network models transform physical systems into latent Gaussian distributions.

Datasets such as images, text, or movies are embedded in high-dimensional spaces. However, in important cases such as images of objects, the statistical structure in the data constrains samples to a manifold of dramatically lower dimensionality. Learning to identify and extract task-relevant variables from this embedde…

In this paper we develop a novel computational sensing framework for sensing and recovering structured signals. When trained on a set of representative signals, our framework learns to take undersampled measurements and recover signals from them using a deep convolutional neural network. In other words, it learns a tra…

An algorithmic limit of compressed sensing or related variable-selection problems is analytically evaluated when a design matrix is given by an overcomplete random matrix. The replica method from statistical mechanics is employed to derive the result. The analysis is conducted through evaluation of the entropy, an expo…

We describe the multi-GPU gradient boosting algorithm implemented in the XGBoost library (https://github.com/dmlc/xgboost). Our algorithm allows fast, scalable training on multi-GPU systems with all of the features of the XGBoost library. We employ data compression techniques to minimise the usage of scarce GPU memory …

Study uncovers scaling laws and spectral properties of shallow neural networks.

In this paper, the `Approximate Message Passing' (AMP) algorithm, initially developed for compressed sensing of signals under i.i.d. Gaussian measurement matrices, has been extended to a multi-terminal setting (MAMP algorithm). It has been shown that similar to its single terminal counterpart, the behavior of MAMP algo…

Study on theoretical limits of $\ell_0$ sparse-regression algorithms using Fl RDT.