arXiv research

Finet uses FBN for efficient, lightweight neural networks.

We present a method for feature interpretation that makes use of recent advances in autoregressive density estimation models to invert model representations. We train generative inversion models to express a distribution over input features conditioned on intermediate model representations. Insights into the invariance…

The Information Bottleneck (IB) objective uses information theory to formulate a task-performance versus robustness trade-off. It has been successfully applied in the standard discriminative classification setting. We pose the question whether the IB can also be used to train generative likelihood models such as normal…

Training Gaussian process-based models typically involves an $O(N^3)$ computational bottleneck due to inverting the covariance matrix. Popular methods for overcoming this matrix inversion problem cannot adequately model all types of latent functions, and are often not parallelizable. However, judicious choice of model…

We explore models for translating abstract musical ideas (scores, rhythms) into expressive performances using Seq2Seq and recurrent Variational Information Bottleneck (VIB) models. Though Seq2Seq models usually require painstakingly aligned corpora, we show that it is possible to adapt an approach from the Generative A…

A new method integrates autoencoders with geometry regularization for manifold learning.

We address representational challenges in normalizing flows, particularly depth and conditioning issues.

New research shows non-bottlenecked autoencoders can outperform bottlenecked ones for anomaly detection.

Study on estimating invertible functions with minimax analysis.

SCBMs model causal effects using low-dimensional bottlenecks.

Local invertibility of higher order tensor transforms on compact manifolds.

Study of strongly invertible Legendrian links in contact 3-space.

Softmax is an output activation function for modeling categorical probability distributions in many applications of deep learning. However, a recent study revealed that softmax can be a bottleneck of representational capacity of neural networks in language modeling (the softmax bottleneck). In this paper, we propose an…

Study on invariant Seifert surfaces for strongly invertible knots, showing large gaps in genus.

There has recently been much work on the "wide limit" of neural networks, where Bayesian neural networks (BNNs) are shown to converge to a Gaussian process (GP) as all hidden layers are sent to infinite width. However, these results do not apply to architectures that require one or more of the hidden layers to remain n…

Dirac operator invertibility proven for specific manifolds.

Table of symmetric diagrams for knots up to 10 crossings.

ISR creates analytical relationships from data via invertible maps.

Study shows bottlenecks improve image segmentation quality.

Local invertibility of ray transforms on convex manifolds.

Deep ResNets favor low bottleneck rank with proper hyperparameters.

This work tackles exploding inverses in INNs, revealing and mitigating their numerical non-invertibility.

CF-INNs can approximate any invertible function, resolving a long-standing problem.

This work presents a novel approach to train invertible linear layers by adding rank-one perturbations.

Develops equivariant grid homology for strongly invertible knots.

CB-APM uses analyst consensus as a bottleneck to interpret stock returns.

New findings on knot genera using advanced techniques.

Defines knot signature invariant using G-signature theorem.

Paper shows invertibility of tensor X-ray transform on certain manifolds.

Deep neural networks are vulnerable to adversarial attacks and hard to interpret because of their black-box nature. The recently proposed invertible network is able to accurately reconstruct the inputs to a layer from its outputs, thus has the potential to unravel the black-box model. An invertible network classifier c…

Wide neural networks become linear, but adding bottlenecks makes them bilinear or multilinear.

For operators of many different kinds it has been proved that (generalized) Darboux transformations can be built using so called Wronskian formulae. Such Darboux transformations are not invertible in the sense that the corresponding mappings of the operator kernels are not invertible. The only known invertible ones wer…

New invertible transformations improve flow-based generative models.

We show that standard ResNet architectures can be made invertible, allowing the same model to be used for classification, density estimation, and generation. Typically, enforcing invertibility requires partitioning dimensions or restricting network architectures. In contrast, our approach only requires adding a simple …

New method quantifies redundant information using information bottleneck.

ButterflyFlow uses butterfly matrices for efficient invertible layers in normalizing flows.

By using parity arguments we prove that free knots are, generally, not invertible.

Electronic power inverters are capable of quickly delivering reactive power to maintain customer voltages within operating tolerances and to reduce system losses in distribution grids. This paper proposes a systematic and data-driven approach to determine reactive power inverter output as a function of local measuremen…

Many recent invertible neural architectures are based on coupling block designs where variables are divided in two subsets which serve as inputs of an easily invertible (usually affine) triangular transformation. While such a transformation is invertible, its Jacobian is very sparse and thus may lack expressiveness. Th…

Flow-based generative models parameterize probability distributions through an invertible transformation and can be trained by maximum likelihood. Invertible residual networks provide a flexible family of transformations where only Lipschitz conditions rather than strict architectural constraints are needed for enforci…

Invertible DenseNets improve model efficiency and performance.

Two knots with unique surgery properties.

We propose a new way of constructing invertible neural networks by combining simple building blocks with a novel set of composition rules. This leads to a rich set of invertible architectures, including those similar to ResNets. Inversion is achieved with a locally convergent iterative procedure that is parallelizable …

By a result of John Ball (1981), a locally orientation preserving Sobolev map is almost everywhere globally invertible whenever its boundary values admit a homeomorphic extension. As shown here for any dimension, the conclusions of Ball's theorem and related results can be reached while completely avoiding the problem …

New spectral sequences define knot invariants.

The Mahler volume of a centrally symmetric convex body K is defined as M(K)= (Vol K)(Vol K^dual). Mahler conjectured that this volume is minimized when K is a cube. We introduce the bottleneck conjecture, which stipulates that a certain convex body K^diamond subset K X K^dual has least volume when K is an ellipsoid. If…

The Information bottleneck method is an unsupervised non-parametric data organization technique. Given a joint distribution P(A,B), this method constructs a new variable T that extracts partitions, or clusters, over the values of A that are informative about B. The information bottleneck has already been applied to doc…

Neural ODEs and i-ResNet are recently proposed methods for enforcing invertibility of residual neural models. Having a generic technique for constructing invertible models can open new avenues for advances in learning systems, but so far the question of whether Neural ODEs and i-ResNets can model any continuous inverti…