arXiv research

The paper compares unrolling and bilevel optimization for learning variational models.

PES method reduces bias in gradient estimation for unrolled graphs.

Statistical analysis of algorithm unrolling for inverse problems.

This work analyzes the convergence rate of unrolling for optimizing quadratic objectives.

ES-Single uses ES to estimate gradients in unrolled graphs, reducing variance and improving performance.

Recurrent convolution (RC) shares the same convolutional kernels and unrolls them multiple steps, which is originally proposed to model time-space signals. We argue that RC can be viewed as a model compression strategy for deep convolutional neural networks. RC reduces the redundancy across layers. However, the perform…

This work improves dictionary learning speed without sacrificing accuracy.

Unrolled networks learn optimal Bayesian inference for unknown priors.

PUDLE method analyzes and improves unrolled sparse coding networks for dictionary learning.

New neural network improves MRI reconstruction for non-Cartesian data.

Modified Hennings invariant defined using quantum groups and integrals.

Unrolled neural networks emerged recently as an effective model for learning inverse maps appearing in image restoration tasks. However, their generalization risk (i.e., test mean-squared-error) and its link to network design and train sample size remains mysterious. Leveraging the Stein's Unbiased Risk Estimator (SURE…

While neural networks have achieved vastly enhanced performance over traditional iterative methods in many cases, they are generally empirically designed and the underlying structures are difficult to interpret. The algorithm unrolling approach has helped connect iterative algorithms to neural network architectures. Ho…

Blind image deblurring remains a topic of enduring interest. Learning based approaches, especially those that employ neural networks have emerged to complement traditional model based methods and in many cases achieve vastly enhanced performance. That said, neural network approaches are generally empirically designed a…

The paper develops generalization bounds for deep compound Gaussian neural networks.

We show that unrolled quantum groups at odd roots of unity give rise to relative modular categories. These are the main building blocks for the construction of 1+1+1-TQFTs extending CGP invariants, which are non-semisimple quantum invariants of closed 3-manifolds decorated with ribbon graphs and cohomology classes. Whe…

This paper introduces Non-Autonomous Input-Output Stable Network(NAIS-Net), a very deep architecture where each stacked processing block is derived from a time-invariant non-autonomous dynamical system. Non-autonomy is implemented by skip connections from the block input to each of the unrolled processing stages and al…

We introduce a method to stabilize Generative Adversarial Networks (GANs) by defining the generator objective with respect to an unrolled optimization of the discriminator. This allows training to be adjusted between using the optimal discriminator in the generator's objective, which is ideal but infeasible in practice…

Quantum groups give lower genus bounds for links.

In this paper, we propose an end-to-end deep learning model, called E2Efold, for RNA secondary structure prediction which can effectively take into account the inherent constraints in the problem. The key idea of E2Efold is to directly predict the RNA base-pairing matrix, and use an unrolled algorithm for constrained p…

This paper accelerates TV regularization algorithms by unrolling proximal gradient descent.

Transformers interpreted as probabilistic Laplacian Eigenmaps steps.

uGLAD recovers sparse graphs from data using deep unrolled networks.

We introduce an architecture based on deep hierarchical decompositions to learn effective representations of large graphs. Our framework extends classic R-decompositions used in kernel methods, enabling nested part-of-part relations. Unlike recursive neural networks, which unroll a template on input graphs directly, we…

LASE learns graph embeddings by unrolling GD iterations into a neural network.

We develop the general theory for the construction of Extended Topological Quantum Field Theories (ETQFTs) associated with the Costantino-Geer-Patureau quantum invariants of closed 3-manifolds. In order to do so, we introduce relative modular categories, a class of ribbon categories which are modeled on representations…

We prove the unrolled superalgebra $\mathcal{U}_ξ^{H}\mathfrak{sl}(2|1)$ has a completion which is a ribbon superalgebra in a topological sense where $ξ$ is a root of unity of odd order. Using this ribbon superalgebra we construct its universal invariant of links. We use it to construct an invariant of $3$-manifolds of…

Careful tuning of the learning rate, or even schedules thereof, can be crucial to effective neural net training. There has been much recent interest in gradient-based meta-optimization, where one tunes hyperparameters, or even learns an optimizer, in order to minimize the expected loss when the training procedure is un…

Enhanced ECCD speeds up elastic net model training.

Linear encoding of sparse vectors is widely popular, but is commonly data-independent -- missing any possible extra (but a priori unknown) structure beyond sparsity. In this paper we present a new method to learn linear encoders that adapt to data, while still performing well with the widely used $\ell_1$ decoder. The …

Model-based Reinforcement Learning approaches have the promise of being sample efficient. Much of the progress in learning dynamics models in RL has been made by learning models via supervised learning. But traditional model-based approaches lead to `compounding errors' when the model is unrolled step by step. Essentia…

A Hermitian TQFT from non-semisimple quantum sl(2) modules.

Many challenging image processing tasks can be described by an ill-posed linear inverse problem: deblurring, deconvolution, inpainting, compressed sensing, and superresolution all lie in this framework. Traditional inverse problem solvers minimize a cost function consisting of a data-fit term, which measures how well a…

We develop a diagrammatic calculus for representations of unrolled quantum $\mathfrak{sl}_2$ at a fourth root of unity. This allows us to prove Seifert-Torres type formulas for certain splice links using quantum algebraic methods, rather than topological methods. Other applications of this diagrammatic calculus given h…

In this work, we propose a deep neural network architecture motivated by primal-dual splitting methods from convex optimization. We show theoretically that there exists a close relation between the derived architecture and residual networks, and further investigate this connection in numerical experiments. Moreover, we…

Invariants of 3-manifolds from a non semi-simple category of modules over a version of quantum $sl(2)$ were obtained by the last three authors in arXiv:1202.3553 . They are invariants of $3$-manifolds together with a cohomology class which can be interpreted as a line bundle with flat connection. In arXiv:1404.7289 we …

Accelerates Birkhoff projection for manifold-constrained hyper-connections with high accuracy and speed.

Because of their effectiveness in broad practical applications, LSTM networks have received a wealth of coverage in scientific journals, technical blogs, and implementation guides. However, in most articles, the inference formulas for the LSTM network and its parent, RNN, are stated axiomatically, while the training fo…

Given a set of observations generated by an optimization process, the goal of inverse optimization is to determine likely parameters of that process. We cast inverse optimization as a form of deep learning. Our method, called deep inverse optimization, is to unroll an iterative optimization process and then use backpro…

Bayesian neural networks learn graph structure with interpretable parameters.

Deep residual networks (ResNets) and their variants are widely used in many computer vision applications and natural language processing tasks. However, the theoretical principles for designing and training ResNets are still not fully understood. Recently, several points of view have emerged to try to interpret ResNet …

We study relationships between the restricted unrolled quantum group $\overline{U}_q^H(\mathfrak{sl}_2)$ at $2r$-th root of unity $q=e^{πi/r}, r \geq 2$, and the singlet vertex operator algebra $\mathcal M(r)$. We use deformable families of modules to efficiently compute $(1, 1)$-tangle invariants colored with projecti…

ASTRA improves TDA by more accurately approximating iHVP.

This work improves uncertainty estimates for LISTA estimators.

Memory-efficient learning for large-scale imaging systems.

Image reconstruction from undersampled k-space data has been playing an important role for fast MRI. Recently, deep learning has demonstrated tremendous success in various fields and also shown potential to significantly speed up MR reconstruction with reduced measurements. This article gives an overview of deep learni…

Learning a policy using only observational data is challenging because the distribution of states it induces at execution time may differ from the distribution observed during training. We propose to train a policy by unrolling a learned model of the environment dynamics over multiple time steps while explicitly penali…

The study identifies conditions under which algorithmic stability explains generalization in interpolating learning systems.