arXiv research

Binary encoding enables neural networks to extrapolate periodic functions.

Note: Causality can be encoded without strict time function choice.

Paper explores neural network approximations on sphere domains.

Despite the superior performance of deep learning in many applications, challenges remain in the area of regression on function spaces. In particular, neural networks are unable to encode function inputs compactly as each node encodes just a real value. We propose a novel idea to address this shortcoming: to encode an …

What do auto-encoders learn about the underlying data generating distribution? Recent work suggests that some auto-encoder variants do a good job of capturing the local manifold structure of data. This paper clarifies some of these previous observations by showing that minimizing a particular form of regularized recons…

Quantum models can approximate any function if data encoding allows for a rich enough frequency spectrum.

We present a framework to learn privacy-preserving encodings of images that inhibit inference of chosen private attributes, while allowing recovery of other desirable information. Rather than simply inhibiting a given fixed pre-trained estimator, our goal is that an estimator be unable to learn to accurately predict th…

We propose computationally efficient encoders and decoders for lossy compression using a Sparse Regression Code. The codebook is defined by a design matrix and codewords are structured linear combinations of columns of this matrix. The proposed encoding algorithm sequentially chooses columns of the design matrix to suc…

Improves energy efficiency of neuromorphic hardware by optimizing memory organization and encoding schemes.

MetaFun learns functional representations for meta-learning.

We introduce a novel encoder-decoder architecture to embed functional processes into latent vector spaces. This embedding can then be decoded to sample the encoded functions over any arbitrary domain. This autoencoder generalizes the recently introduced Conditional Neural Process (CNP) model of random processes. Our ar…

A simple encoder and complex decoder for secure image encryption and decryption.

Recent work suggests that some auto-encoder variants do a good job of capturing the local manifold structure of the unknown data generating density. This paper contributes to the mathematical understanding of this phenomenon and helps define better justified sampling algorithms for deep learning based on auto-encoder v…

Study efficient neural operator learning using variation spaces.

Hybrid Bayesian neural networks use function uncertainty for probabilistic inference.

New framework infers multiple classes per image for one-shot learning.

Stochastic encoding improves gender classification of brain networks from UK Biobank data.

We present a new approach to 3D object representation where a neural network encodes the geometry of an object directly into the weights and biases of a second 'mapping' network. This mapping network can be used to reconstruct an object by applying its encoded transformation to points randomly sampled from a simple geo…

Given a convolutional dictionary underlying a set of observed signals, can a carefully designed auto-encoder recover the dictionary in the presence of noise? We introduce an auto-encoder architecture, termed constrained recurrent sparse auto-encoder (CRsAE), that answers this question in the affirmative. Given an input…

New method encodes function preferences into neural nets for better generalization.

New method for scalable set encoding with unbiased gradient approximation.

We describe a method of encoding various types of link diagrams, including those with classical, flat, rigid, welded, and virtual crossings. We show that this method may be used to encode link diagrams, up to equivalence, in a notation whose length is a cubic function of the number of 'riser marks'. For classical knots…

GraphToken encodes structured data for LLMs, improving graph reasoning tasks.

The paper investigates instance-based interpretation methods for VAEs.

MCSAE improves speaker embedding by focusing on both high- and low-level features.

Despite its nonconvex nature, $\ell_0$ sparse approximation is desirable in many theoretical and application cases. We study the $\ell_0$ sparse approximation problem with the tool of deep learning, by proposing Deep $\ell_0$ Encoders. Two typical forms, the $\ell_0$ regularized problem and the $M$-sparse problem, are …

New insights into encoder-decoder structures using information measures.

New bounds for quantum circuits depend on how data is encoded.

Traditional set prediction models can struggle with simple datasets due to an issue we call the responsibility problem. We introduce a pooling method for sets of feature vectors based on sorting features across elements of the set. This can be used to construct a permutation-equivariant auto-encoder that avoids this re…

This paper explores GNN functions on random graphs, highlighting the importance of node Positional Encodings.

Tensor networks and RNNs are equivalent, improving wave function encoding.

Study presents a method to induce a generalized neural network from joint group invariant functions.

Novel Bayesian prior for neural networks encodes amplitude and lengthscale.

Generative adversarial networks (GANs) have demonstrated to be successful at generating realistic real-world images. In this paper we compare various GAN techniques, both supervised and unsupervised. The effects on training stability of different objective functions are compared. We add an encoder to the network, makin…

The paper uses neural networks to forecast time series data.

We show that a generative random field model, which we call generative ConvNet, can be derived from the commonly used discriminative ConvNet, by assuming a ConvNet for multi-category classification and assuming one of the categories is a base category generated by a reference distribution. If we further assume that the…

A novel approach stores encoded images as centroids and covariance matrices to improve classification accuracy with less memory.

Brain networks in fMRI are typically identified using spatial independent component analysis (ICA), yet mathematical constraints such as sparse coding and positivity both provide alternate biologically-plausible frameworks for generating brain networks. Non-negative Matrix Factorization (NMF) would suppress negative BO…

Novel PCA method for high-dimensional inverse problems.

VAEs analyzed using harmonic analysis, showing how variance controls frequency content and robustness.

We consider an online decision making setting known as contextual bandit problem, and propose an approach for improving contextual bandit performance by using an adaptive feature extraction (representation learning) based on online clustering. Our approach starts with an off-line pre-training on unlabeled history of co…

Diversity plays a vital role in many text generating applications. In recent years, Conditional Variational Auto Encoders (CVAE) have shown promising performances for this task. However, they often encounter the so called KL-Vanishing problem. Previous works mitigated such problem by heuristic methods such as strengthe…

Constrained clustering has been well-studied for algorithms such as $K$-means and hierarchical clustering. However, how to satisfy many constraints in these algorithmic settings has been shown to be intractable. One alternative to encode many constraints is to use spectral clustering, which remains a developing area. I…

Efficiently accelerates attention calculation for Transformers with relative positional encoding.

We construct a deep portfolio theory. By building on Markowitz's classic risk-return trade-off, we develop a self-contained four-step routine of encode, calibrate, validate and verify to formulate an automated and general portfolio selection process. At the heart of our algorithm are deep hierarchical compositions of p…

We present a novel architecture, the "stacked what-where auto-encoders" (SWWAE), which integrates discriminative and generative pathways and provides a unified approach to supervised, semi-supervised and unsupervised learning without relying on sampling during training. An instantiation of SWWAE uses a convolutional ne…

Enhances neural models with simple functions to improve language modeling.

Null distance encodes causal structure in spacetimes.