Flow-based generative models (Dinh et al., 2014) are conceptually attractive due to tractability of the exact log-likelihood, tractability of exact latent-variable inference, and parallelizability of both training and synthesis. In this paper we propose Glow, a simple type of generative flow using an invertible 1x1 con…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “invertible 1x1 convolutions”
Incorporates matrix exponential into generative flows for improved performance.
problem Improving generative flow models for better density estimation.
method Integrates matrix exponential into generative flows, proposing new layers and modifying network architecture.
result The proposed model achieves great performance on density estimation.
Woodbury transformations improve deep generative models with efficient invertibility and determinant calculation.
problem Efficiently invertible and determinant-calculable functions for deep generative models.
method Introducing Woodbury transformations that leverage matrix identities for efficient invertibility and determinant calculation.
result Woodbury transformations enable high-dimensional interactions, efficient sampling, and likelihood evaluation, outperforming other flow architectures.
ButterflyFlow uses butterfly matrices for efficient invertible layers in normalizing flows.
problem Building efficient invertible layers for complex probability distributions.
method Proposes butterfly layers for invertible linear layers, leveraging their ability to capture complex structures.
result ButterflyFlow achieves strong density estimation and significantly better log-likelihoods on various datasets.
Paper proposes multiscale self-attentive convolutions for vision and language.
problem Improving language and vision understanding models using self-attention.
method Developed 1D and 2D Self Attentive Convolutions (SAC), multiscale SAC (MSAC).
result MSAC enhances model performance for vision and language tasks.
Predicting Length of Stay in the Intensive Care Unit with Temporal Pointwise Convolutional Networkscs.LG
New model predicts ICU patients' stay duration efficiently.
problem Efficient ICU bed allocation under resource constraints.
method Temporal Pointwise Convolution (TPC) model combining temporal and pointwise convolutions.
result Significant performance improvements over LSTM and Transformer models.
Generative flows are attractive because they admit exact likelihood optimization and efficient image synthesis. Recently, Kingma & Dhariwal (2018) demonstrated with Glow that generative flows are capable of generating high quality images. We generalize the 1 x 1 convolutions proposed in Glow to invertible d x d convolu…
New linear flows using exponential of linear transformations improve generative models.
problem Improving generative models in machine learning.
method Developed convolution exponentials and generalized Sylvester Flows using the exponential of linear transformations.
result Convolution exponentials and Convolutional Sylvester Flows outperform other models in log-likelihood.
A new KD layer lets student models learn and apply teacher knowledge explicitly.
problem Implicit action of traditional KD on student's feature transform limits its use in intermediate layers.
method Proposes a learnable KD layer that explicitly embeds teacher's knowledge in feature transform.
result Improves KD with two abilities: leveraging teacher's knowledge and feeding forward knowledge deeper.
Several recent works have empirically observed that Convolutional Neural Nets (CNNs) are (approximately) invertible. To understand this approximate invertibility phenomenon and how to leverage it more effectively, we focus on a theoretical explanation and develop a mathematical model of sparse signal recovery that is c…
SparseRT accelerates sparse computations on GPUs for deep learning inference.
problem Efficiently handling unstructured sparsity patterns on GPUs for deep learning.
method SparseRT, a code generator that leverages unstructured sparsity for accelerating sparse linear algebra operations.
result Geometric mean speedups of 3.4x at 90% sparsity and 5.4x at 95% sparsity for 1x1 convolutions and fully connected layers.
Temporal Pointwise Convolutional Networks for Length of Stay Prediction in the Intensive Care Unitcs.LG
New model predicts ICU patient stays more accurately.
problem Efficient ICU bed allocation under resource constraints.
method Temporal Pointwise Convolutional Networks (TPC) combining temporal and pointwise convolutions.
result Significant performance improvements over LSTM and Transformer models.
Compared with global average pooling in existing deep convolutional neural networks (CNNs), global covariance pooling can capture richer statistics of deep features, having potential for improving representation and generalization abilities of deep CNNs. However, integration of global covariance pooling into deep CNNs …
We propose a new way of constructing invertible neural networks by combining simple building blocks with a novel set of composition rules. This leads to a rich set of invertible architectures, including those similar to ResNets. Inversion is achieved with a locally convergent iterative procedure that is parallelizable …
Finet uses FBN for efficient, lightweight neural networks.
problem Building efficient neural networks with limited computational resources.
method Introduces Fine-grained Batch Normalization (FBN) and a novel light-weight network (Finet) that combines FBN with standard convolution.
result Finet achieves state-of-the-art performance on ImageNet classification with reduced computational complexity.
When deep learning models on GPU can be accelerated by taking advantage of unstructured sparsitycs.LG
This paper accelerates sparse CNN layers on GPUs by using unstructured sparsity.
problem Efficiency of sparse CNN layers on GPUs.
method Direct sparse operation and reduced precision.
result Achieving up to 90% sparsity in deep CNN models improves efficiency.
Residual neural networks don't help overcome sampling complexity issues.
problem Learning invertible residual neural networks from samples is hard due to the curse of dimensionality.
method Investigated invertible residual neural networks and their sampling complexity.
result Invertible residual neural networks still suffer from the curse of dimensionality in sampling complexity.
Deep adversarial neural decodingq-bio.NC
Here, we present a novel approach to solve the problem of reconstructing perceived stimuli from brain responses by combining probabilistic inference with deep learning. Our approach first inverts the linear transformation from latent features to brain responses with maximum a posteriori estimation and then inverts the …
Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEscs.LG
We present a simple neural rendering architecture that helps variational autoencoders (VAEs) learn disentangled representations. Instead of the deconvolutional network typically used in the decoder of VAEs, we tile (broadcast) the latent vector across space, concatenate fixed X- and Y-"coordinate" channels, and apply a…
iGNN tackles inverse graph prediction using invertible neural networks.
problem Inverse graph prediction problem in data analysis and machine learning.
method Developed invertible graph neural network (iGNN) to solve inverse prediction problem on graphs.
result iGNN model allows efficient generation from output labels and forward prediction.
The paper explores how invertibility affects the complexity of encoder models in VAEs.
problem The complexity of the encoder model in VAEs when the generative map is invertible.
method Formalizes the concept of strong invertibility and analyzes the complexity of the encoder model.
result Strongly invertible generative maps allow for simpler encoder models, while non-invertible maps require exponentially larger encoders.
This research studies affine invariance in continuous-domain convolutional neural networks.
problem Recognizing patterns and features under affine transformations in continuous domains.
method Introduces a new criterion for assessing affine invariance, embeds images into the affine Lie group, and analyzes convolution over this group.
result Extends the scope of geometrical transformations that deep-learning pipelines can handle.
Deep neural network compression techniques such as pruning and weight tensor decomposition usually require fine-tuning to recover the prediction accuracy when the compression ratio is high. However, conventional fine-tuning suffers from the requirement of a large training set and the time-consuming training procedure. …
Majority of semantic segmentation algorithms operate on a single frame even in the case of videos. In this work, the goal is to exploit temporal information within the algorithm model for leveraging motion cues and temporal consistency. We propose two simple high-level architectures based on Recurrent FCN (RFCN) and Mu…
It is widely believed that the success of deep convolutional networks is based on progressively discarding uninformative variability about the input with respect to the problem at hand. This is supported empirically by the difficulty of recovering images from their hidden representations, in most commonly used network …
Flowification enriches neural networks with an inverse pass and likelihood monitoring.
problem Neural networks lack an inverse pass and likelihood monitoring, limiting their generative capabilities.
method Introduce flowification, enriching neural networks with a stochastic inverse pass and likelihood monitoring.
result Certain neural network architectures can be enriched to fall under the generalized notion of a normalizing flow.
A Bayesian procedure is developed for multivariate stochastic volatility, using state space models. An autoregressive model for the log-returns is employed. We generalize the inverted Wishart distribution to allow for different correlation structure between the observation and state innovation vectors and we extend the…
A major issue in harmonic analysis is to capture the phase dependence of frequency representations, which carries important signal properties. It seems that convolutional neural networks have found a way. Over time-series and images, convolutional networks often learn a first layer of filters which are well localized i…
In this article, we start to recall the inversion formula for the convolution with the Box spline. The equivariant cohomology and the equivariant K-theory with respect to a compact torus G of various spaces associated to a linear action of G in a vector space M can be both described using some vector spaces of distribu…
We present a method for feature interpretation that makes use of recent advances in autoregressive density estimation models to invert model representations. We train generative inversion models to express a distribution over input features conditioned on intermediate model representations. Insights into the invariance…
Paper aims to find joint representation between vocal tract geometry and speech sound acoustics.
problem Finding a joint latent representation between articulatory and acoustic domains for vowel sounds.
method Invertible neural network models, convolutional autoencoder, normalizing flows, semi-supervised learning.
result Satisfactory performance in articulatory-to-acoustic and acoustic-to-articulatory mapping.
Paper investigates Lipschitz constants of self-attention modules in neural networks.
problem Lipschitz constants of self-attention modules in neural networks.
method Proved standard dot-product self-attention is not Lipschitz for unbounded input domain. Proposed L2 self-attention that is Lipschitz. Derived upper bound on L2 self-attention's Lipschitz constant.
result Proved standard self-attention is not Lipschitz for unbounded input domain and proposed an alternative L2 self-attention that is Lipschitz.
Study on estimating invertible functions with minimax analysis.
problem Minimizing risk of estimating invertible functions on a plane.
method Introduce two types of -risks, derive lower and upper rates for minimax values, develop an asymptotically almost everywhere invertible estimator.
result Invertibility does not reduce the complexity of the estimation problem in terms of the rate.
Generative Adversarial Nets (GANs) and Variational Auto-Encoders (VAEs) provide impressive image generations from Gaussian white noise, but the underlying mathematics are not well understood. We compute deep convolutional network generators by inverting a fixed embedding operator. Therefore, they do not require to be o…
Local invertibility of higher order tensor transforms on compact manifolds.
problem Invertibility of higher order tensor transforms on compact manifolds.
method Local invertibility of transverse and mixed ray transforms of tensors on compact Riemannian manifolds.
result Local invertibility of transverse and mixed ray transforms of tensors for specific dimensions.
Generative compression technique reduces neural network size and improves performance on microcontrollers.
problem Memory constraints on microcontrollers limit the size of neural networks, especially for 1x1 pointwise (PW) mixers.
method HYPER-TINYPW uses a shared micro-MLP to generate PW kernels from tiny per-layer codes, reducing memory usage.
result HYPER-TINYPW achieves comparable performance to larger models while being significantly smaller (225 kB vs 1.4 MB).
Study of strongly invertible Legendrian links in contact 3-space.
problem Characterizing and understanding strongly invertible Legendrian links.
method Equivariant analogs of basic results for strongly invertible and Legendrian links.
result Existence of maximal equivariant Thurston-Bennequin number for strongly invertible links.
We address representational challenges in normalizing flows, particularly depth and conditioning issues.
problem Challenges in training normalizing flows, including vanishing/exploding gradients and poor conditioning.
method Analyzes representational aspects of depth and conditioning in normalizing flows, proving theoretical bounds and investigating phenomena.
result Proves that shallow affine coupling networks are universal approximators in Wasserstein distance if ill-conditioning is allowed.
Study on invariant Seifert surfaces for strongly invertible knots, showing large gaps in genus.
problem Understanding gaps in genus between strongly invertible knots and their invariant Seifert surfaces.
method Analysis of invariant Seifert surfaces and proof of genus gaps, with variants of Edmonds' theorem.
result Gap between equivariant genus and usual genus can be arbitrarily large for strongly invertible knots.
Dirac operator invertibility proven for specific manifolds.
problem Invertibility of twisted Dirac operator on manifolds.
method Closed connected spin manifold with non-negative scalar curvature, flat Hilbert module bundle.
result Dirac operator is invertible under given conditions.
Table of symmetric diagrams for knots up to 10 crossings.
problem Finding symmetric diagrams for strongly invertible knots.
method Compilation of symmetric diagrams for knots up to 10 crossings.
result Similarity of transversal diagrams to symmetric union diagrams for strongly invertible knots.
Global invertibility proven for orientation-preserving maps without homeomorphic extension.
problem Global invertibility of orientation-preserving Sobolev maps.
method Avoiding homeomorphic extension, study of strictly orientation-preserving maps.
result Global invertibility can be achieved without homeomorphic extension.
ISR creates analytical relationships from data via invertible maps.
problem Creating analytical relationships from datasets.
method Combines INNs and EQL, using invertible maps and sparsity promoting regularization.
result ISR can serve as a normalizing flow for density estimation and solve inverse problems.
Local invertibility of ray transforms on convex manifolds.
problem Invertibility of ray transforms on compact Riemannian manifolds with strictly convex boundary.
method Local invertibility results for transverse and mixed ray transforms of 1 and 1+1 tensors.
result Local invertibility of ray transforms near boundary points, leading to global results.
This work tackles exploding inverses in INNs, revealing and mitigating their numerical non-invertibility.
problem Exploding inverses in INNs cause numerical non-invertibility, leading to failures in various tasks.
method Derived bi-Lipschitz properties of INN building blocks, proposed regularizers for local invertibility, and stable INN designs for global invertibility.
result Bi-Lipschitz properties and stable INN designs are crucial for addressing numerical non-invertibility.
CF-INNs can approximate any invertible function, resolving a long-standing problem.
problem Whether CF-INNs can approximate any invertible function.
method Demonstrated CF-INNs are universal approximators for invertible functions by showing a convenient criterion.
result CF-INNs are universal approximators for invertible functions.
This work presents a novel approach to train invertible linear layers by adding rank-one perturbations.
problem Training invertible linear layers during optimization with gradient-based methods is challenging.
method Train rank-one perturbations and add them to weight matrices infrequently, keeping track of inverses and determinants.
result Invertible linear layers improve mixing and mode separation in normalizing flows.
Develops equivariant grid homology for strongly invertible knots.
problem Invariants of strongly invertible knots.
method Equivariant grid diagrams and mapping cones.
result Equivariant unknotting numbers and genus bounds.