arXiv research

Unsupervised representation learning via generative modeling is a staple to many computer vision applications in the absence of labeled data. Variational Autoencoders (VAEs) are powerful generative models that learn representations useful for data generation. However, due to inherent challenges in the training objectiv…

Paper shows regularization improves robustness in domain generalization.

Regularized LAEs learn principal components efficiently.

Autoencoders and their variations provide unsupervised models for learning low-dimensional representations for downstream tasks. Without proper regularization, autoencoder models are susceptible to the overfitting problem and the so-called posterior collapse phenomenon. In this paper, we introduce a quantization-based …

New autoencoder learns structured representations without regularization.

Researchers compute determinants and torsions of Rumin complex in specific Lie group representations.

We prove that any minimal (maximal) strongly regular surface in the three-dimensional Minkowski space locally admits canonical principal parameters. Using this result, we find a canonical representation of minimal strongly regular time-like surfaces, which makes more precise the Weierstrass representation and shows mor…

AL2 progressively penalizes network activations to prevent overfitting.

Unified theory for representation learning using learnable functions.

Statistical characteristics of deep network representations, such as sparsity and correlation, are known to be relevant to the performance and interpretability of deep learning. When a statistical characteristic is desired, often an adequate regularizer can be designed and applied during the training phase. Typically, …

A method identifies domain-general features using causal graph constraints and regularization.

We identify action representations from video data, proving their statistical benefits.

In this study, the effects of eight representation regularization methods are investigated, including two newly developed rank regularizers (RR). The investigation shows that the statistical characteristics of representations such as correlation, sparsity, and rank can be manipulated as intended, during training. Furth…

SinSim improves self-supervised learning by integrating optimal transport into contrastive learning.

Group equivariant and steerable convolutional neural networks (regular and steerable G-CNNs) have recently emerged as a very effective model class for learning from signal data such as 2D and 3D images, video, and other data where symmetries are present. In geometrical terms, regular G-CNNs represent data in terms of s…

Study on how optimal representations emerge during deep learning training, focusing on the role of implicit regularization.

Statistical methods protecting sensitive information or the identity of the data owner have become critical to ensure privacy of individuals as well as of organizations. This paper investigates anonymization methods based on representation learning and deep neural networks, and motivated by novel information theoretica…

We systematically investigate the problem of representing Markov chains by families of random maps, and which regularity of these maps can be achieved depending on the properties of the probability measures. Our key idea is to use techniques from optimal transport to select optimal such maps. Optimal transport theory a…

A new method uses algebraic insights to create approximately equivariant networks without complex architectures.

We provide adaptive inference methods, based on $\ell_1$ regularization, for regular (semi-parametric) and non-regular (nonparametric) linear functionals of the conditional expectation function. Examples of regular functionals include average treatment effects, policy effects, and derivatives. Examples of non-regular f…

The paper calculates Alexander polynomials for knots using finite group representations.

Spectral embedding is a popular technique for the representation of graph data. Several regularization techniques have been proposed to improve the quality of the embedding with respect to downstream tasks like clustering. In this paper, we explain on a simple block model the impact of the complete graph regularization…

We give sufficient conditions for a parametrised family of probability measures on a Riemannian manifold with boundary to be represented by random maps of class $C^k$. The conditions allow for the probability densities to approach zero towards the boundary of the manifold. We also formulate two obstructions to regular …

Large neural networks learn low-dimensional representations that balance complexity and regularity.

There has been an increased interest in multimodal language processing including multimodal dialog, question answering, sentiment analysis, and speech recognition. However, naturally occurring multimodal data is often imperfect as a result of imperfect modalities, missing entries or noise corruption. To address these c…

Training deep neural networks is known to require a large number of training samples. However, in many applications only few training samples are available. In this work, we tackle the issue of training neural networks for classification task when few training samples are available. We attempt to solve this issue by pr…

Represents neural networks as solutions to inverse problems in Banach spaces.

New method for high-fidelity shape representations from raw data.

New method uses random convex polytopes to measure representation quality.

The standard approach for dealing with the ill-posedness of the training problem in machine learning and/or the reconstruction of a signal from a limited number of measurements is regularization. The method is applicable whenever the problem is formulated as an optimization task. The standard strategy consists in augme…

Sparsity inducing regularization is an important part for learning over-complete visual representations. Despite the popularity of $\ell_1$ regularization, in this paper, we investigate the usage of non-convex regularizations in this problem. Our contribution consists of three parts. First, we propose the leaky capped …

It is proved that every discrete Morse function in the sense of Forman on a finite regular CW complex can be represented by a polyhedral Morse function in the sense of Banchoff on an appropriate embedding in Euclidean space of the barycentric subdivision of the CW complex; such a representation preserves critical point…

Multi-head attention mechanism is capable of learning various representations from sequential data while paying attention to different subsequences, e.g., word-pieces or syllables in a spoken word. From the subsequences, it retrieves richer information than a single-head attention which only summarizes the whole sequen…

New technique prevents Q-learning collapse by maximizing diversity among ensembles.

Unified taxonomy for graph representation learning.

Variational Autoencoder (VAE), a simple and effective deep generative model, has led to a number of impressive empirical successes and spawned many advanced variants and theoretical investigations. However, recent studies demonstrate that, when equipped with expressive generative distributions (aka. decoders), VAE suff…

Two definitions quantify $C^{2,α}$ regularity of Riemannian surfaces.

New framework improves reliability of learned representations by modeling uncertainty and structural constraints.

New representation theory for closed geodesic subflows.

A VB-groupoid is a Lie groupoid equipped with a compatible linear structure. In this paper, we describe a correspondence, up to isomorphism, between VB-groupoids and 2-term representations up to homotopy of Lie groupoids. Under this correspondence, the tangent bundle of a Lie groupoid G corresponds to the "adjoint repr…

A generative model with a disentangled representation allows for independent control over different aspects of the output. Learning disentangled representations has been a recent topic of great interest, but it remains poorly understood. We show that even for GANs that do not possess disentangled representations, one c…

Given a flag in each of the vertex-transitive tessellations of the Euclidean plane by regular polygons, we determine the flag stabilizer under the action of the automorphism group of a regular cover. In so doing we give a presentation of these tilings as quotients of regular (infinite) polyhedra.

FairNN learns fair representations and decisions by optimizing a multi-objective loss function.

A focal representation of a generic regular curve γ in E^{m+1} consists of the centers of the osculating hyperplanes. A k-slant helix γ in E^{m+1} is a (generic) regular curve whose unit normal vector V_{k} makes a constant angle with a fixed direction U in E^{m+1}. In the present paper we proved that if γ is a k-slant…

RO-TD learns sparse value functions efficiently.

Sequential learning, also called lifelong learning, studies the problem of learning tasks in a sequence with access restricted to only the data of the current task. In this paper we look at a scenario with fixed model capacity, and postulate that the learning process should not be selfish, i.e. it should account for fu…

With the rapid development of social media sharing, people often need to manage the growing volume of multimedia data such as large scale video classification and annotation, especially to organize those videos containing human activities. Recently, manifold regularized semi-supervised learning (SSL), which explores th…

Modern deep neural networks require a tremendous amount of data to train, often needing hundreds or thousands of labeled examples to learn an effective representation. For these networks to work with less data, more structure must be built into their architectures or learned from previous experience. The learned weight…