This work proposes a new method to train models with deep latent hierarchies using Optimal Transport.
problem Training models with deep latent hierarchies using VAEs often leads to the 'latent variable collapse' issue.
method Proposes a novel approach based on Optimal Transport to train models with deep latent hierarchies.
result The method avoids the 'latent variable collapse' issue and provides better sample generations and latent representation.
New method for deep learning hierarchies like sequences and graphs.
problem Lack of formalism for dealing with hierarchical structures in deep learning.
method Express equivariant maps for hierarchies using combinations of building block symmetries.
result Demonstrates effectiveness on semantic segmentation of point-cloud data.
BIVA uses a deep hierarchy of latent variables for better generative modeling.
problem Performance gap between VAE and autoregressive models in generative modeling.
method BIVA introduces a skip-connected generative model and a bidirectional stochastic inference path.
result BIVA reaches state-of-the-art test likelihoods and generates coherent images.
Deep networks learn hierarchical data by invariant representations.
problem How many examples are needed for deep networks to learn hierarchical data?
method Random Hierarchy Model: synthetic tasks inspired by language and images hierarchy.
result Deep networks learn by invariant representations and require a detectable number of correlations between low-level features and classes.
Proves deep networks can learn hierarchical structures efficiently.
problem Understanding how deep networks learn hierarchical structures in data.
method Random Hierarchy Models, gradient-based methods, layerwise training.
result Proves deep networks can efficiently learn hierarchical structures.
SRHM explains deep learning's hierarchy and insensitivity to transformations.
problem Understanding how deep networks learn hierarchical and invariant representations.
method Introducing sparsity to generative hierarchical models of data.
result Hierarchical representations and insensitivity to transformations correlate strongly with deep network performance.
Introduces a probabilistic view of deep learning for better understanding and explaining neural networks.
problem Explaining the behavior and properties of deep neural networks.
method Introduces a probabilistic representation of deep learning, linking neurons, hidden layers, and the whole architecture to Gibbs distributions and Bayesian neural networks.
result Demonstrates the hierarchy and generalization properties of deep learning through a probabilistic lens.
Study of deep neural networks' NTK evolution during training.
problem Understanding the performance gap between deep neural networks and kernel regression.
method Derive an infinite hierarchy of ordinary differential equations (NTH) to capture gradient descent dynamics of deep neural networks.
result Truncated NTH approximates the dynamic of the NTK up to arbitrary precision under certain conditions.
Deep Boltzmann machines are in principle powerful models for extracting the hierarchical structure of data. Unfortunately, attempts to train layers jointly (without greedy layer-wise pretraining) have been largely unsuccessful. We propose a modification of the learning algorithm that initially recenters the output of t…
A network supporting deep unsupervised learning is presented. The network is an autoencoder with lateral shortcut connections from the encoder to decoder at each level of the hierarchy. The lateral shortcut connections allow the higher levels of the hierarchy to focus on abstract invariant features. While standard auto…
ProHOC detects OOD samples in class hierarchies, predicting them to correct internal nodes.
problem Binary OOD detection ignores semantic relationships between OOD and ID classes.
method Probabilistic hierarchical model using multi-depth networks trained for ID classification.
result ProHOC effectively classifies OOD samples to their correct internal nodes in class hierarchies.
New framework explains how deep networks reduce complexity and maintain weak correlations.
problem Understanding the interpretability of deep neural networks.
method Mean-field framework to analyze deterministic and generative deep networks.
result Deep computation reduces dimensionality while keeping weak neuron correlations.
New method improves blackbox attack transferability by perturbing feature hierarchy.
problem Improving transferability of blackbox attacks across different models and datasets.
method Perturbs representations throughout feature hierarchy to mimic other classes.
result Achieves 10x increase in targeted success rate compared to other methods.
New deep architecture models uncertainty by sharing neural connectivity patterns.
problem Uncertainty modeling in deep neural networks remains challenging.
method Proposes a new deep architecture that shares neural connectivity patterns between generative and discriminative networks to model a confounder.
result Demonstrates significant improvement in uncertainty estimation compared to state-of-the-art methods.
The study analyzes games and social hierarchies, incorporating luck and depth of competition.
problem Analyzing patterns of wins and losses in games and social hierarchies.
method Generalized probabilistic models incorporating luck and depth of competition.
result Social competition tends to be deeper with many distinct levels, but there is often a chance of upset victories.
A new algorithm for bandits with hierarchical rewards.
problem Learning from correlated rewards in complex hierarchies.
method Hierarchical Thompson Sampling (HierTS) for Gaussian hierarchies.
result Hierarchical Thompson Sampling reduces regret by non-constant factors in the number of actions.
WHAI combines autoencoding and MCMC for scalable topic modeling.
problem Training scalable deep topic models for big corpora.
method Develops a hybrid autoencoding inference network for deep latent Dirichlet allocation.
result WHAI achieves both scalability and speed in topic inference.
In this note we present a generative model of natural images consisting of a deep hierarchy of layers of latent random variables, each of which follows a new type of distribution that we call rectified Gaussian. These rectified Gaussian units allow spike-and-slab type sparsity, while retaining the differentiability nec…
New method reduces memory usage in deep HRNNs by replacing gradient backpropagation with local losses.
problem Memory constraints in training deep hierarchical RNNs.
method Replace gradient backpropagation with locally computable losses in deep HRNNs.
result Memory requirements reduced by a factor exponential in hierarchy depth.
Concept Hierarchies and Formal Concept Analysis are theoretically well grounded and largely experimented methods. They rely on line diagrams called Galois lattices for visualizing and analysing object-attribute sets. Galois lattices are visually seducing and conceptually rich for experts. However they present important…
Proposes a new multi-layer model for topic distributions.
problem Leveraging deep structures for learning word distributions of topics.
method A multi-layer generative process on word distributions of topics, where each topic is drawn from a mixture of topics from the layer above.
result Discover interpretable topic hierarchies and improve topic models' accuracy and interpretability.
Hierarchical nucleation patterns emerge in deep neural network layers.
problem Understanding the generation of meaningful representations in deep neural networks.
method Analysis of the probability density of ImageNet dataset across hidden layers.
result Density peaks in subsequent layers mirror the semantic hierarchy of concepts, resembling nucleation process.
Proposes a neural network for accurate and reconciled hierarchical time series forecasting.
problem Forecasting and reconciling hierarchical time series data.
method Uses a deep neural network to directly produce accurate and reconciled forecasts, minimizing a customized loss function at training time.
result Our approach outperforms state-of-the-art competitors in hierarchical forecasting on real-world datasets.
HCRL learns hierarchical embeddings from deep embeddings of hierarchy components.
problem Flat clustering limits cohesive instance relations in hierarchical data.
method Simultaneously optimizes representation learning and hierarchical clustering in the embedding space.
result HCRL achieves best hierarchical clustering and data reconstruction.
This work extends the options framework to learn at multiple temporal resolutions.
problem Autonomous creation of temporal abstractions from data in reinforcement learning.
method Developed a hierarchical option-critic architecture capable of learning at multiple temporal resolutions.
result Derived policy gradient theorems for a deep hierarchy of options.
Constructs integrable hierarchies for generalized Frobenius manifolds with non-flat unity.
problem Integrable hierarchies for generalized Frobenius manifolds with non-flat unity.
method Constructs a bihamiltonian integrable hierarchy of hydrodynamic type.
result Integrable hierarchy possesses Virasoro symmetries and a tau structure.
Paper studies ResNet dynamics using NTH, reducing width requirement.
problem Understanding ResNet dynamics and improving training efficiency.
method Uses Neural Tangent Hierarchy (NTH) to analyze ResNet dynamics.
result Reduces width requirement from quartic to cubic for ResNet.
A new multilevel framework speeds up ResNet training.
problem Training deep residual networks (ResNets) is time-consuming.
method Formulates ResNets as dynamical systems and uses time-dependent optimal control problems.
result Enhanced training of ResNets with multilevel auxiliary networks achieves significant speedup.
Study identifies pitfalls in assessing hierarchies for multi-class classification.
problem Lack of understanding in selecting hierarchies for multi-class classification.
method Analyzed and compared popular approaches to extracting hierarchies.
result Hierarchy quality becomes irrelevant when using powerful classifiers.
Super tau-covers extend bihamiltonian hierarchies' symmetries.
problem Extending symmetries of bihamiltonian hierarchies.
method Constructing super tau-covers for bihamiltonian integrable hierarchies.
result Symmetries of bihamiltonian hierarchies extended to super tau-covers.
We study the problem of topic modeling in corpora whose documents are organized in a multi-level hierarchy. We explore a parametric approach to this problem, assuming that the number of topics is known or can be estimated by cross-validation. The models we consider can be viewed as special (finite-dimensional) instance…
Hydrodynamic hierarchy deformed using conservation laws.
problem Deforming a hydrodynamic hierarchy with non-vanishing Nijenhuis torsion.
method Using a chain of conservation laws to deform the hierarchy.
result The resulting hierarchy has non-vanishing Nijenhuis torsion but vanishing Haantjes tensor.
New hierarchy for a special group type.
problem Classifying relatively hyperbolic virtually special groups.
method Constructing a new virtual quasiconvex hierarchy.
result Generalized Malnormal Special Quotient Theorem.
New hierarchical RL agent learns to generalize in complex multi-agent games.
problem Current RL methods struggle to generalize to unseen opponents in multi-agent games.
method Proposed a hierarchical RL architecture grounded in game-theoretic structure.
result Hierarchical agent generalizes to unseen opponents, while baselines fail.
RICH models scenes as hierarchical tree to learn and generate complex compositions.
problem Learning compositional structures between parts and objects in natural scenes.
method RICH uses a latent scene graph to organize entities into a tree structure and employs a top-down inference approach.
result RICH learns and generates complex scene hierarchies from unlabeled data.
Legendre transformations link related integrable hierarchies.
problem Understanding relationships between integrable hierarchies.
method Legendre-type transformations of generalized Frobenius manifolds.
result Linear reciprocal transformations link related hierarchies.
Twisted U- and twisted U/K-hierarchies are soliton hierarchies introduced by Terng to find higher flows of the generalized sine-Gordon equation. Twisted O(J)×O(J)O(J,J)-hierarchies are among the most important classes of twisted hierarchies. In this paper, interesting first and higher flows of twi…
Bihamiltonian structures lead to tau structures for integrable hierarchies.
problem Classifying deformations of bihamiltonian structures.
method Starting from flat exact semisimple bihamiltonian structures, we derive Frobenius manifolds and tau structures.
result Deformations of the principal hierarchy with tau structures are classified.
Novel method extracts hierarchical brain connectivity patterns from fMRI.
problem Functional hierarchical organization of the human brain.
method Sparse Connectivity Patterns (SCPs) with hierarchy of sparse overlapping patterns, deep factorization of correlation matrices.
result Reproducible multi-scale hierarchical SCPs more stable than single-scale patterns.
Derive bihamiltonian structure for rational reduction of 2D-Toda hierarchy
problem Derive bihamiltonian structure for rational reduction of 2D-Toda hierarchy
method Direct computations
result Derive local bihamiltonian structure
Deep learning approximates system moments from data.
problem Approximating moments of spatial probabilistic systems.
method Dynamic Boltzmann Distributions (DBDs) with deep Boltzmann machines (DBMs).
result Learned moment closures improve generalization over traditional methods.
We propose a formally completely integrable extension of heat hierarchy based on the space of symmetries isomorphic to the Weyl algebra A1. The extended heat hierarchy will be the basic model for the analysis of the extension of KP hierarchy, and other integrable equations.
HyPE improves sample efficiency in DRL by discovering objects and hierarchies of skills.
problem Poor sample efficiency in DRL methods, especially in complex tasks.
method HyPE algorithm that discovers objects and generates hypotheses about their controllability, learning a hierarchy of skills.
result HyPE learns high-scoring policies an order of magnitude faster than state-of-the-art methods.
New hierarchies derived from KP hierarchy using non-formal operators and Yang-Mills action.
problem Formal solutions of KP hierarchy and their non-formal counterparts.
method Developed new hierarchies of non-linear equations on non-formal pseudo-differential operators.
result Expressed one hierarchy as Yang-Mills action minimization.
We introduce a deep, generative autoencoder capable of learning hierarchies of distributed representations from data. Successive deep stochastic hidden layers are equipped with autoregressive connections, which enable the model to be sampled from quickly and exactly via ancestral sampling. We derive an efficient approx…
Symmetry reduction of Painlevé IV to Flaschka-Newell Painlevé II
problem Isomonodromic deformation problem associated with rank-two meromorphic connections
method Symmetry Ψ(−λ)=σ1Ψ(λ)σ1 result Induced isomonodromic dynamics coincides with Flaschka-Newell Painlevé II hierarchy
Proposes a new learning mechanism for deep networks using local errors.
problem Learning using delayed and non-local errors is hard for biological neural networks.
method Generates local errors in each layer using fixed, random auxiliary classifiers.
result Approaches standard backpropagation performance on MNIST, CIFAR10, and SVHN datasets.
Scroll structures on solutions of 4D integrable equations are involutive and governed by a dispersionless hierarchy.
problem Characterizing the geometry of solutions to 4D integrable equations.
method Defining rational normal scrolls and showing their involutivity.
result Involutive scroll structures are governed by a dispersionless integrable hierarchy.