arXiv research

Geodesics connect model modes in neural network loss landscapes.

Proposes using mode connectivity to improve adversarial robustness of neural networks.

Neural networks can learn optimal auction mechanisms and satisfy mode connectivity.

Proposes neuron alignment to optimize mode connectivity in neural networks.

Periodic surfaces have a limited number of bending modes, equal to their membrane modes.

Revisits zero modes of Dirac operator on Eguchi-Hanson space.

Mode clustering is a nonparametric method for clustering that defines clusters using the basins of attraction of a density estimator's modes. We provide several enhancements to mode clustering: (i) a soft variant of cluster assignment, (ii) a measure of connectivity between clusters, (iii) a technique for choosing the …

Paper discovers simplicial complexes connecting trained models for improved ensembling.

We consider the Yang-Mills flow on hyperbolic 3-space. The gauge connection is constructed from the frame-field and (not necessarily compatible) spin connection components. The fixed points of this flow include zero Yang-Mills curvature configurations, for which the spin connection has zero torsion and the associated R…

The paper reveals surprising star-shaped connectivity in neural networks.

We identify and study two common failure modes for early training in deep ReLU nets. For each we give a rigorous proof of when it occurs and how to avoid it, for fully connected and residual architectures. The first failure mode, exploding/vanishing mean activation length, can be avoided by initializing weights from a …

Multimodal clustering is an unsupervised technique for mining interesting patterns in $n$-adic binary relations or $n$-mode networks. Among different types of such generalized patterns one can find biclusters and formal concepts (maximal bicliques) for 2-mode case, triclusters and triconcepts for 3-mode case, closed $n…

Generative adversarial networks (GANs) are innovative techniques for learning generative models of complex data distributions from samples. Despite remarkable recent improvements in generating realistic images, one of their major shortcomings is the fact that in practice, they tend to produce samples with little divers…

Many generative models have to combat $\textit{missing modes}$. The conventional wisdom to this end is by reducing through training a statistical distance (such as $f$-divergence) between the generated distribution and provided data distribution. But this is more of a heuristic than a guarantee. The statistical distanc…

Mode connectivity is a recently introduced frame- work that empirically establishes the connected- ness of minima by finding a high accuracy curve between two independently trained models. To investigate the limits of this setup, we examine the efficacy of this technique in extreme cases where the input models are trai…

Improves conditions for mode connectivity in deep neural networks.

Study on catenoid stability using asymmetric potentials.

The loss functions of deep neural networks are complex and their geometric properties are not well understood. We show that the optima of these complex loss functions are in fact connected by simple curves over which training and test accuracy are nearly constant. We introduce a training procedure to discover these hig…

This paper characterizes VAE training pathologies and their effects on tasks.

This work builds the connection between the regularity theory of optimal transportation map, Monge-Ampère equation and GANs, which gives a theoretic understanding of the major drawbacks of GANs: convergence difficulty and mode collapse. According to the regularity theory of Monge-Ampère equation, if the support of the …

LMC loss barrier decreases to zero with large network width.

Sampling from posterior distributions using Markov chain Monte Carlo (MCMC) methods can require an exhaustive number of iterations, particularly when the posterior is multi-modal as the MCMC sampler can become trapped in a local mode for a large number of iterations. In this paper, we introduce the pseudo-extended MCMC…

Unified theory explains two failure modes of deep transformers and provides initialisation guidelines.

The use of imitation learning to learn a single policy for a complex task that has multiple modes or hierarchical structure can be challenging. In fact, previous work has shown that when the modes are known, learning separate policies for each mode or sub-task can greatly improve the performance of imitation learning. …

The paper uses RMT to analyze global banking network changes since 2000.

LoRA-Curve connects independent LoRA optima through continuous low-loss valleys, improving Bayesian model averaging.

Paper proposes ManiF-SMC for effective approximate machine unlearning.

Sparse-mode DMD disambiguates local and global modes in spatiotemporal data.

Generative adversarial networks (GANs) are the state of the art in generative modeling. Unfortunately, most GAN methods are susceptible to mode collapse, meaning that they tend to capture only a subset of the modes of the true distribution. A possible way of dealing with this problem is to use an ensemble of GANs, wher…

Transformers infer tasks from context via two modes, geometrically shaped task vectors explain their behavior.

Motivated by advantages of current-mode design, this brief contribution explores the implementation of weight matrices in neuromemristive systems via current-mode memristor crossbar circuits. After deriving theoretical results for the range and distribution of weights in the current-mode design, it is shown that any we…

The convergence rate and final performance of common deep learning models have significantly benefited from heuristics such as learning rate schedules, knowledge distillation, skip connections, and normalization layers. In the absence of theoretical underpinnings, controlled experiments aimed at explaining these strate…

Introduces a massive variant of Ray-Singer Torsion to avoid zero modes in topological field theories.

New method for Bayesian neural networks reduces inference difficulty.

Recent work on mode connectivity in the loss landscape of deep neural networks has demonstrated that the locus of (sub-)optimal weight vectors lies on continuous paths. In this work, we train a neural network that serves as a hypernetwork, mapping a latent vector into high-performance (low-loss) weight vectors, general…

For dynamical systems that can be modelled as asymptotically stable linear systems forced by Gaussian noise, this paper develops methods to infer or estimate their modes from observations in real time. The modes can be real or complex. For a real mode, we wish to infer its damping rate and mode shape. For a complex mod…

Unified framework connects different neural network models.

Characterizes neutral deformation modes of minimal surfaces.

Mathematical analysis shows annealing prevents mode collapse in Gaussian mixtures.

New method renormalizes neural network Gaussian processes to identify learnable vs. unlearnable modes.

Parsimonious Dynamic Mode Decomposition selects sparse modes robustly.

This study applies EMD to MSCI World index and converts IMFs into graphs for GNN modeling.

Deep learning has seen tremendous success over the past decade in computer vision, machine translation, and gameplay. This success rests in crucial ways on gradient-descent optimization and the ability to learn parameters of a neural network by backpropagating observed errors. However, neural network architectures are …

VINNAS uses variational inference to avoid mode collapse in neural architecture search.

A new, efficient $k$-modes algorithm improves clustering of categorical data.

This paper studies the space of $L ^2$ harmonic forms and $L ^2$ harmonic spinors on Taub-bolt, a Ricci-flat Riemannian 4-manifold of ALF type. We prove that the space of harmonic square-integrable 2-forms on Taub-bolt is 2-dimensional and construct a basis. We explicitly find a 2-parameter family of $L ^2$ zero mod…

Improved modeling of chaotic systems using time-delay embeddings and Frenet-Serret frame.

EDLP samples flat modes in discrete spaces using entropy.