In this paper we address the following question, given a face representation, how many identities can it resolve? In other words, what is the capacity of the face representation? A scientific basis for estimating the capacity of a given face representation will not only benefit the evaluation and comparison of differen…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “Representational Capacity”
Researchers analyze neural process architectures and their representational capacities.
problem Understanding what functions can be represented by different neural process architectures.
method Analyzing four types of neural process architectures: CNPs, ANPs, TNPs, and their latent variants.
result Prove these architectures form a strict hierarchy and characterize their representational capabilities.
Paper applies theorem to find optimal investment boundary in stochastic capacity expansion.
problem Finding optimal investment boundary in a stochastic, time-inhomogeneous capacity expansion problem.
method Applies Bank and El Karoui Representation Theorem to solve first order conditions involving a non-integral term.
result Existence of base capacity , showing optimal investment process becomes active at this level.
gLSTM improves graph neural networks by increasing storage capacity to prevent over-squashing.
problem Over-squashing in GNNs collapses information from a large receptive field into a single vector, creating an information bottleneck.
method Introduced a new synthetic task to measure over-squashing and adapted ideas from sequence modeling to develop gLSTM, a novel GNN architecture with improved capacity.
result gLSTM architecture demonstrates strong performance on synthetic and real-world graph benchmarks, mitigating over-squashing.
We extend manifold capacity to nonlinear neural representations with contextual information.
problem Efficient processing of information through neural representations.
method Theoretical framework leveraging latent directions in input space related to contextual information.
result Derivation of an exact formula for context-dependent manifold capacity.
The brain optimizes memory by forgetting what's predictable, improving generalization.
problem Memory consolidation struggles with representational drift, semanticisation, and offline replay.
method Proposes predictive forgetting as a mechanism to optimize generalization by reducing complexity.
result Predictive forgetting improves information-theoretic generalization bounds on stored representations.
Memory capacity of DAM scales exponentially with feature separation, unaffected by correlations.
problem Understanding how feature correlations impact DAM's capacity.
method Developed an empirical framework to analyze DAM's capacity under varying feature correlations and pattern separations.
result Memory capacity scales exponentially with feature separation, unaffected by correlations.
Semi-supervised node classification in attributed graphs, i.e., graphs with node features, involves learning to classify unlabeled nodes given a partially labeled graph. Label predictions are made by jointly modeling the node and its' neighborhood features. State-of-the-art models for node classification on such attrib…
There is some theoretical evidence that deep neural networks with multiple hidden layers have a potential for more efficient representation of multidimensional mappings than shallow networks with a single hidden layer. The question is whether it is possible to exploit this theoretical advantage for finding such represe…
Study shows how correlations between neural activity affect classification capacity.
problem Understanding how correlations between neural activity impact classification performance.
method Calculated the capacity of neural activity on spherical manifolds with and without correlations between centroids and axes.
result Introducing correlations between neural activity centroids pushes spheres closer together, while correlations between axes shrink their radii, revealing a duality between correlations and geometry in classification.
Proposes a new graph representation method using tensor products.
problem Dynamic graph representation and theoretical properties.
method Bind-and-sum approach in hyperdimensional computing (HDC), tensor product as binding operation.
result Memory vs. size analysis of graph representation size scaling.
Recurrent neural networks are powerful models for processing sequential data, but they are generally plagued by vanishing and exploding gradient problems. Unitary recurrent neural networks (uRNNs), which use unitary recurrence matrices, have recently been proposed as a means to avoid these issues. However, in previous …
We study a stochastic, continuous time model on a finite horizon for a firm that produces a single good. We model the production capacity as an Ito diffusion controlled by a nondecreasing process representing the cumulative investment. The firm aims to maximize its expected total net profit by choosing the optimal inve…
New analysis tightens memory capacity of Hopfield models using spherical codes.
problem Optimizing memory capacity in modern Hopfield models and Kernelized Hopfield Models.
method Connecting Hopfield models to spherical codes in information theory, establishing an optimal capacity bound and a sub-linear algorithm.
result First tight and optimal asymptotic memory capacity for modern Hopfield models, matching known lower bounds.
Bézier-GAN optimizes airfoil design by reducing shape complexity.
problem High computational cost in aerodynamic shape optimization.
method Generative adversarial networks (GANs) to learn compact shape representations.
result Empirically accelerates optimization convergence by at least two times.
SOC-ICNN expands neural network representational capacity by using conic optimization.
problem Restrictive representational capacity of ReLU-based ICNNs.
method Proposes SOC-ICNN architecture that uses Second-Order Cone Programming.
result SOC-ICNN strictly expands representational space without increasing complexity.
This paper focuses on the discrimination capacity of aggregation functions: these are the permutation invariant functions used by graph neural networks to combine the features of nodes. Realizing that the most powerful aggregation functions suffer from a dimensionality curse, we consider a restricted setting. In partic…
Study on risk measures using distorted Choquet integrals with random distortions.
problem Developing risk measures under random distortions of capacities.
method Introducing and analyzing randomly distorted Choquet integrals with respect to a distorted capacity, establishing properties and providing representations.
result Representation of comonotonic additive conditional risk measures using G-randomly distorted Choquet integrals.
Generative autoencoders offer a promising approach for controllable text generation by leveraging their latent sentence representations. However, current models struggle to maintain coherent latent spaces required to perform meaningful text manipulations via latent vector operations. Specifically, we demonstrate by exa…
This paper investigates how data augmentation improves linear separation of manifold data.
problem Understanding how data augmentation enhances linear separation of manifold data.
method Investigates the conditions under which self-supervised representations can linearly separate multi-manifold data.
result Self-supervised learning can linearly separate manifolds with a smaller distance than unsupervised learning.
A grand challenge in representation learning is to learn the different explanatory factors of variation behind the high dimen- sional data. Encoder models are often determined to optimize performance on training data when the real objective is to generalize well to unseen data. Although there is enough numerical eviden…
Model for cross-border markets with limited transmission capacities.
problem Limited transmission capacities between two countries' markets.
method Developed a regime-switching process model with high-frequency approximation.
result Analytic tractability allows computation of key market quantities.
With the advent of large labelled datasets and high-capacity models, the performance of machine vision systems has been improving rapidly. However, the technology has still major limitations, starting from the fact that different vision problems are still solved by different models, trained from scratch or fine-tuned o…
Sparse codes improve optimal control tasks with correlated inputs.
problem Optimal control tasks with correlated feature inputs.
method Used a sparse code to represent natural images in an optimal control task solved with neuro-dynamic programming.
result An over-complete sparse code increases memory capacity and learning speed beyond a complete code.
Simple object representations improve model-free RL performance.
problem Current reinforcement learning agents lack object recognition.
method Used simple, feature-engineered object representations with the Rainbow model.
result Object representations significantly boost performance on Atari games.
We obtain a dual representation of the Kantorovich functional defined for functions on the Skorokhod space using quotient sets. Our representation takes the form of a Choquet capacity generated by martingale measures satisfying additional constraints to ensure compatibility with the quotient sets. These sets contain st…
Wide neural networks can degrade performance, contrary to conventional wisdom.
problem Understanding the limitations of increasing network width in neural networks.
method Using Deep Gaussian Processes to decouple capacity and width, analyzing their effects on representational power and non-Gaussianity.
result Wide neural networks can become less adaptable and more Gaussian, leading to performance degradation.
We present new intuitions and theoretical assessments of the emergence of disentangled representation in variational autoencoders. Taking a rate-distortion theory perspective, we show the circumstances under which representations aligned with the underlying generative factors of variation of data emerge when optimising…
Holomorphic networks on modular arithmetic show clear success or failure, no in-between.
problem Understanding when neural networks can represent modular arithmetic tasks.
method Two-layer networks with holomorphic monomial activations trained on modular tasks.
result The network's output is confined to a subspace of characters, and representability depends on the task's Fourier support.
GANs can improve image reconstruction by using intermediate layers.
problem Improving the quality of image reconstruction using GANs.
method Exploiting the representation in intermediate layers of the generator.
result Intermediate layers in GANs can represent natural images with high visual fidelity.
Paper investigates Lambda Value-at-Risk under ambiguity and risk sharing.
problem Investigates Lambda Value-at-Risk under ambiguity and risk sharing.
method Establishes equivalence of robust VaR and traditional VaR under ambiguity sets, analyzes properties, derives explicit formulas, and explores risk sharing.
result Unified and extended the concept of Value-at-Risk under ambiguity, derived explicit formulas for specific ambiguity sets, and explored risk sharing.
Linear Readout of Object Manifoldscond-mat.dis-nn
Objects are represented in sensory systems by continuous manifolds due to sensitivity of neuronal responses to changes in physical features such as location, orientation, and intensity. What makes certain sensory representations better suited for invariant decoding of objects by downstream networks? We present a theory…
The problem of high-dimensional and large-scale representation of visual data is addressed from an unsupervised learning perspective. The emphasis is put on discrete representations, where the description length can be measured in bits and hence the model capacity can be controlled. The algorithmic infrastructure is de…
This paper extends financial theory to measure learnable market structure under computational constraints.
problem Understanding learnable market structure under bounded computational capacity.
method Introduces financial epiplexity as a measure of learnable market structure, extending classical information theory.
result Proves that equal entropy does not imply equal epiplexity and derives thresholds for useful regimes.
Intelligent behaviour in the real-world requires the ability to acquire new knowledge from an ongoing sequence of experiences while preserving and reusing past knowledge. We propose a novel algorithm for unsupervised representation learning from piece-wise stationary visual data: Variational Autoencoder with Shared Emb…
Selfless Sequential Learningstat.ML
Sequential learning, also called lifelong learning, studies the problem of learning tasks in a sequence with access restricted to only the data of the current task. In this paper we look at a scenario with fixed model capacity, and postulate that the learning process should not be selfish, i.e. it should account for fu…
IRMAE learns compact latent spaces by minimizing rank.
problem Learning compact latent representations in autoencoders.
method Implicitly minimizes the rank of the covariance matrix through gradient descent in multi-layer linear networks.
result Demonstrates validity on image generation and representation learning tasks.
We address the problem of one-to-many mappings in supervised learning, where a single instance has many different solutions of possibly equal cost. The framework of conditional variational autoencoders describes a class of methods to tackle such structured-prediction tasks by means of latent variables. We propose to in…
Representation learning has become an invaluable approach for learning from symbolic data such as text and graphs. However, while complex symbolic datasets often exhibit a latent hierarchical structure, state-of-the-art methods typically learn embeddings in Euclidean vector spaces, which do not account for this propert…
The recently developed variational autoencoders (VAEs) have proved to be an effective confluence of the rich representational power of neural networks with Bayesian methods. However, most work on VAEs use a rather simple prior over the latent variables such as standard normal distribution, thereby restricting its appli…
Characterizes continuity of monotone functionals in mixed topology.
problem Continuity of monotone functionals in mixed topology.
method Characterization through lower semicontinuity and dual representations.
result Continuity in mixed topology is equivalent to dual representation in terms of countably additive measures.
New Feedback Transformer architecture improves model performance by exposing past representations to future.
problem Limitations of Transformers in fully exploiting sequential input.
method Proposes Feedback Transformer exposing all past representations to future.
result Demonstrates improved performance with smaller, shallower models.
Understanding the representational power of Restricted Boltzmann Machines (RBMs) with multiple layers is an ill-understood problem and is an area of active research. Motivated from the approach of \emph{Inherent Structure formalism} (Stillinger & Weber, 1982), extensively used in analysing Spin Glasses, we propose a no…
Proves generalization bounds for SGD using Feller processes and Hausdorff dimension.
problem Characterizing generalization properties of SGD in deep learning.
method Proves generalization bounds for SGD under Feller process approximation, linking generalization error to the Hausdorff dimension of trajectories.
result Generalization error controlled by the Hausdorff dimension of trajectories, which is linked to the tail behavior of the driving process.
Variational methods that rely on a recognition network to approximate the posterior of directed graphical models offer better inference and learning than previous methods. Recent advances that exploit the capacity and flexibility in this approach have expanded what kinds of models can be trained. However, as a proposal…
We address the problem of communicating domain knowledge from a user to the designer of a clustering algorithm. We propose a protocol in which the user provides a clustering of a relatively small random sample of a data set. The algorithm designer then uses that sample to come up with a data representation under which …
New analysis explains pathology of deep Gaussian processes.
problem Pathology of deep Gaussian processes reduces learning capacities with increased layers.
method Study nonlinear dynamic systems corresponding to DGPs, derive recurrence relations.
result Provide tighter bounds and rate of convergence for dynamic systems.
We study various capacities on compact Kähler manifolds which generalize the Bedford-Taylor Monge-Ampère capacity. We then use these capacities to study the existence and the regularity of solutions of complex Monge-Ampère equations.