PES method reduces bias in gradient estimation for unrolled graphs.
problem High variance and bias in gradient estimation for unrolled computation graphs.
method Divide graph into unrolls, apply ES update, accumulate correction terms.
result PES provides unbiased, low-variance gradient estimates.
Statistical analysis of algorithm unrolling for inverse problems.
problem Designing deep neural networks to solve inverse problems efficiently.
method Analysis of gradient descent network (GDN) unrolling depth and statistical performance.
result The optimal statistical performance of GDNs requires unrolling depth of order log(n)/log(ρ_n^-1), where ρ_n is the convergence rate.
ES-Single uses ES to estimate gradients in unrolled graphs, reducing variance and improving performance.
problem Estimating gradients in unrolled computation graphs with low variance and stability.
method Evolution strategies (ES) applied to unrolled graphs, with a single perturbation per particle.
result ES-Single reduces variance compared to PES, leading to better performance in various tasks.
This work improves dictionary learning speed without sacrificing accuracy.
problem Prohibitive computational cost of standard dictionary learning methods.
method Approximate dictionary learning using unrolling and gradient descent.
result Unrolling outperforms standard methods in support estimation and early iterations.
This work analyzes the convergence rate of unrolling for optimizing quadratic objectives.
problem The challenge of accurately computing Jacobians through optimization.
method Non-asymptotic convergence-rate analysis of unrolled differentiation for gradient descent and Chebyshev method.
result There is a trade-off between fast asymptotic convergence and immediate but slower convergence due to the learning rate.
This paper accelerates TV regularization algorithms by unrolling proximal gradient descent.
problem Solving Total Variation (TV) regularized problems with iterative algorithms.
method Unrolling proximal gradient descent solvers to learn their parameters.
result Two approaches to compute derivatives through proximal operators improve performance.
PUDLE method analyzes and improves unrolled sparse coding networks for dictionary learning.
problem Dictionary learning problem, representing data as a combination of few atoms.
method PUDLE method addresses challenges in unrolled sparse coding networks through theoretical analysis and practical strategies.
result PUDLE method provides conditions for recovering and preserving the support of the latent code, and resolves bias and instability issues.
Deep network learns image deblurring from iterative algorithms.
problem Blind image deblurring using traditional methods.
method Algorithm unrolling for neural networks.
result Deep Unrolling for Blind Deblurring (DUBLID) outperforms state-of-the-art methods.
New method learns compressed sensing matrix exploiting data structure.
problem Missing structure beyond sparsity in data.
method Gradient unrolling to learn a compressed sensing measurement matrix.
result Significantly fewer measurements for excellent reconstructions.
New deep learning model improves image deblurring with interpretability.
problem Improving image deblurring performance with neural networks.
method Unrolling iterative algorithm to create a neural network architecture.
result Our deep network outperforms state-of-the-art methods in image deblurring.
The paper compares unrolling and bilevel optimization for learning variational models.
problem Learning variational models in supervised learning.
method Analyzes unrolling and bilevel optimization approaches for variational models.
result Unrolling can be better than bilevel optimization, but performance depends on parameters.
Guided Evolutionary Strategies uses surrogate gradients to improve optimization.
problem Optimizing functions with unknown true gradients but available surrogate gradients.
method Combines random search with a search distribution elongated along surrogate gradient directions.
result Improves optimization performance over standard evolutionary strategies and first-order methods.
New quantum invariants derived from unrolled quantum groups match existing Hennings invariants.
problem Constructing non-semisimple quantum invariants for 3-manifolds.
method Using unrolled quantum groups at odd roots of unity and small quantum groups.
result Renormalized Hennings invariants coincide with new quantum invariants.
This paper analyzes the generalization risk of unrolled neural networks using Stein's Unbiased Risk Estimator.
problem Analyzing the generalization risk of unrolled neural networks and its relationship to network design and train sample size.
method Using Stein's Unbiased Risk Estimator (SURE), the paper analyzes the generalization risk with bias and variance components for recurrent unrolled networks, focusing on the degrees-of-freedom (DOF) component and the trace of the end-to-end network Jacobian.
result DOF is well-approximated by the weighted path sparsity of the network under incoherence conditions on the trained weights, and DOF increases with train sample size and converges to the generalization risk for both recurrent and non-recurrent schemes.
Unrolled networks learn optimal Bayesian inference for unknown priors.
problem Optimizing Bayesian inference when the prior is unknown.
method Unrolling neural networks to simulate iterations of inference algorithms.
result Unrolled networks approximate convergence to optimal denoisers for product priors.
RC reduces neural network redundancy and improves performance through independent BN layers.
problem Improving neural network performance and reducing redundancy.
method Recurrent convolution with independent batch normalization layers for different unrolling steps.
result The proposed method improves RC networks' performance and achieves cost-adjustable inference.
New neural network improves MRI reconstruction for non-Cartesian data.
problem Improving MRI reconstruction for non-Cartesian data acquisitions.
method Density-compensated unrolled neural networks.
result Density-compensated unrolled neural networks outperform baselines.
Modified Hennings invariant defined using quantum groups and integrals.
problem Defining a modified Hennings invariant using quantum groups.
method Topological ribbon Hopf algebra, discrete Fourier transforms, symmetrized graded integral, modified trace.
result Modified graded Hennings invariant defined and extended to empty manifolds.
LASE learns graph embeddings by unrolling GD iterations into a neural network.
problem Learning graph embeddings from graph inputs.
method Truncating and interpreting GD iterations as layers in a GNN.
result LASE outperforms eigendecomposition routines and GNNs in link prediction and node classification.
E2Efold predicts RNA secondary structures better than previous methods.
problem RNA secondary structure prediction with constraints.
method End-to-end deep learning model using unrolled algorithms to enforce constraints.
result E2Efold predicts significantly better structures, especially for pseudoknotted structures.
The paper develops generalization bounds for deep compound Gaussian neural networks.
problem Developing theoretical guarantees for the performance of deep neural networks.
method Novel generalization error bounds using a compound Gaussian prior and Dudley's integral.
result Theoretical bounds show generalization error scales O ( n ln ( n ) ) \mathcal{O}(n\sqrt{\ln(n)}) O ( n ln ( n ) ) in signal dimension and O ( ( N e t w o r k S i z e ) 3 / 2 ) \mathcal{O}((Network Size)^{3/2}) O (( N e tw or k S i z e ) 3/2 ) in network size. FDS tackles long horizon hyperparameter optimization issues.
problem Memory scaling and gradient degradation in long horizon tasks.
method Forward-mode differentiation with sharing (FDS).
result Significantly outperforms greedy gradient-based alternatives.
ASTRA improves TDA by more accurately approximating iHVP.
problem Improving insights into training data attribution.
method ASTRA uses EKFAC-preconditioner on Neumann series iterations to accurately approximate iHVP.
result Improving iHVP approximation significantly improves TDA performance.
Enhanced ECCD speeds up elastic net model training.
problem Efficiently solving generalized linear models with elastic net constraints.
method Redesigned cyclic coordinate descent with Taylor expansion and batched computations.
result Empirically shows consistent 3 i m e s 3 imes 3 im es performance improvement over state-of-the-art solvers. We introduce a method to stabilize Generative Adversarial Networks (GANs) by defining the generator objective with respect to an unrolled optimization of the discriminator. This allows training to be adjusted between using the optimal discriminator in the generator's objective, which is ideal but infeasible in practice…
Quantum groups give lower genus bounds for links.
problem Finding lower bounds for Seifert genus of links.
method Using unrolled restricted quantum groups at roots of unity and their invariants.
result ADO link polynomials from quantum groups give genus bounds.
NAIS-Net stabilizes deep networks using non-autonomous dynamical systems.
problem Stabilizing deep neural networks to prevent vanishing/exploding gradients.
method NAIS-Net uses non-autonomous dynamical systems with skip connections to enforce stability.
result NAIS-Net proves to be globally asymptotically stable and reduces generalization gap.
Short-horizon bias causes meta-optimization to favor small learning rates.
problem Short-horizon bias in meta-optimization leads to suboptimal learning rates.
method Analyzes a noisy quadratic cost function and runs meta-optimization experiments on benchmark datasets.
result Meta-optimization chooses too small a learning rate, even with a long time horizon.
Transformers interpreted as probabilistic Laplacian Eigenmaps steps.
problem Improving transformer performance through probabilistic interpretation.
method Probabilistic Laplacian Eigenmaps model derivation and graph diffusion step.
result Subtracting identity from attention matrix improves transformer performance.
UPNs embed planning within a goal-directed policy for effective visuomotor control.
problem Learning abstract representations for visuomotor control and generalization.
method Differentiable planning within a latent space, gradient descent trajectory optimization, end-to-end learning of representations.
result UPNs can transfer visuomotor planning strategies across robots with different morphologies and actuation capabilities.
uGLAD recovers sparse graphs from data using deep unrolled networks.
problem Sparse graph recovery from complex data.
method Optimizing deep unrolled networks to learn precision matrices.
result uGLAD outperforms existing algorithms in sparsity optimization and robustly handles missing data.
We introduce an architecture based on deep hierarchical decompositions to learn effective representations of large graphs. Our framework extends classic R-decompositions used in kernel methods, enabling nested part-of-part relations. Unlike recursive neural networks, which unroll a template on input graphs directly, we…
The paper interprets learned step sizes in deep-unfolded gradient descent.
problem Intuitive interpretation of learned non-constant step sizes in deep-unfolded gradient descent.
method Theoretical analysis and optimization of spectral radius.
result Chebyshev steps achieve the lower bound of convergence rate for first-order methods.
Accelerates Birkhoff projection for manifold-constrained hyper-connections with high accuracy and speed.
problem Inaccurate and slow Birkhoff projection in mHC implementations.
method Dual formulation, Newton's method, implicit differentiation, warp-level CUDA kernel.
result Substantial speedups and accuracy improvements in doubly stochastic projections.
New method trains neural networks to optimize faster than tuned methods.
problem Training learned optimizers is difficult and often leads to poor performance.
method Dynamic weighting of unbiased gradient estimators for a variational loss.
result Trained neural networks optimize faster than tuned first-order methods.
Memory-efficient learning for large-scale imaging systems.
problem Memory limitations in GPUs for real-world large-scale inverse problems.
method Exploits reversibility of network layers to enable data-driven design.
result Demonstrated on small-scale and large-scale real-world systems.
DC3 uses deep learning to solve hard-constrained optimization problems efficiently.
problem Hard constraints in optimization problems make classical solvers slow and infeasible.
method DC3 employs a differentiable procedure to enforce feasibility and unrolls corrections for inequality constraints.
result DC3 achieves near-optimal solutions while maintaining feasibility in both synthetic and real-world tasks.
Explains RNN and LSTM fundamentals, derives formulas, and addresses training issues.
problem Lack of detailed formulas and unrolling techniques in LSTM and RNN literature.
method Derives canonical RNN and LSTM formulas from differential equations, proposes unrolling technique, addresses training difficulties.
result Provides a comprehensive understanding of RNN and LSTM, including detailed formulas and unrolling techniques.
Unified framework for lifted training and inversion of neural networks.
problem Challenges in gradient-based training of deep neural networks.
method Unified framework encapsulating various lifted training strategies.
result Unified framework improves training landscape and stability.
New algorithm for online training of Spiking Neural Networks (SNNs).
problem Training Spiking Neural Networks (SNNs) online with BPTT-equivalent gradients.
method Clear separation of spatial and temporal gradient components, derived from biological insights.
result Online training of SNNs with BPTT-equivalent gradients and low time complexity.
We develop the general theory for the construction of Extended Topological Quantum Field Theories (ETQFTs) associated with the Costantino-Geer-Patureau quantum invariants of closed 3-manifolds. In order to do so, we introduce relative modular categories, a class of ribbon categories which are modeled on representations…
Framework for designing nonlinearities in neural networks with slope constraints.
problem Designing nonlinearities with specific properties for signal processing.
method Variational framework with regularization for slope constraints and optimization of adaptive splines.
result Adaptive nonuniform linear splines achieve global optimum in constrained optimization.
A new deep learning model speeds up MRI by reconstructing from undersampled data.
problem Slow MRI due to undersampling in k-space.
method Unrolling primal-dual hybrid gradient algorithm into a deep network, gradually relaxing constraints.
result Superior MR reconstructions from highly undersampled data.
Paper presents a new method to avoid mode collapse in implicit generative models.
problem Training implicit generative models is challenging due to mode collapse issues.
method Introduces a learning by teaching (LBT) approach to optimize KL-divergence instead of JS-divergence.
result Optimal generator matches the true data distribution, avoiding mode collapse.
OPT framework improves neural network generalization by learning an orthogonal transformation.
problem Improving neural network generalization.
method Orthogonal over-parameterized training (OPT) framework that minimizes hyperspherical energy.
result OPT framework provably minimizes hyperspherical energy and improves empirical generalization.
A Hermitian TQFT from non-semisimple quantum sl(2) modules.
problem Constructing a Hermitian TQFT from a non-semisimple category.
method Endowed a non-semisimple category of quantum sl(2) modules with a Hermitian structure and proved the resulting TQFT is Hermitian.
result Projective representations of the mapping class group in indefinite unitary matrices.
New model clusters graphs using Gromov-Wasserstein discrepancy.
problem Graph clustering with topological structures and node attributes.
method Gromov-Wasserstein discrepancy for relational graph comparison; learns atoms and weights via minimization of discrepancy.
result Model achieves flexible factorization of unaligned graphs with different sizes.
New invariant for 3-manifolds from topological Hopf superalgebra.
problem Constructing an invariant for 3-manifolds.
method Using a ribbon superalgebra from a topological Hopf superalgebra, we construct an invariant of 3-manifolds of Hennings type.
result Constructs a new invariant for 3-manifolds from a topological Hopf superalgebra.