The paper reveals surprising star-shaped connectivity in neural networks.
problem Understanding mode connectivity in neural network landscapes.
method Fine-grained analysis of connectivity in overparameterized and finite minima cases.
result Star-shaped connectivity exists in neural network landscapes, suggesting near convexity.
Neural networks can learn optimal auction mechanisms and satisfy mode connectivity.
problem Optimal auction design in complex settings.
method Generalized RochetNet and affine maximizer auctions.
result Neural networks (RochetNet and generalized version) satisfy mode connectivity.
Geodesics connect model modes in neural network loss landscapes.
problem Connecting modes in neural network loss landscapes.
method Reframed mode connectivity in Information Geometry, hypothesized geodesics as mode-connecting paths, proposed algorithm to approximate geodesics.
result Geodesics achieve mode connectivity in neural networks.
We consider the Yang-Mills flow on hyperbolic 3-space. The gauge connection is constructed from the frame-field and (not necessarily compatible) spin connection components. The fixed points of this flow include zero Yang-Mills curvature configurations, for which the spin connection has zero torsion and the associated R…
LMC loss barrier decreases to zero with large network width.
problem Understanding the LMC phenomenon in neural networks.
method Fine-grained analysis of LMC for two-layer ReLU networks.
result LMC loss barrier decreases to zero at a rate of O(m^-1/2) for large network width.
Proposes using mode connectivity to improve adversarial robustness of neural networks.
problem Improving adversarial robustness of deep neural networks.
method Employing mode connectivity in loss landscapes to study adversarial robustness and propose methods for improvement.
result Path connection learned using limited bonafide data can effectively mitigate adversarial effects while maintaining original accuracy.
For dynamical systems that can be modelled as asymptotically stable linear systems forced by Gaussian noise, this paper develops methods to infer or estimate their modes from observations in real time. The modes can be real or complex. For a real mode, we wish to infer its damping rate and mode shape. For a complex mod…
Review of algorithms for linear system approximations.
problem Linear approximation of high-dimensional dynamical systems.
method State-of-the-art algorithms for low-rank DMD.
result Provides additional details for comprehensive understanding.
Unified framework connects different neural network models.
problem Understanding the geometry of neural network loss landscapes.
method Unified framework capturing four symmetry classes.
result First discovery of low- and zero-barrier linear interpolation paths.
Improved modeling of chaotic systems using time-delay embeddings and Frenet-Serret frame.
problem Identifying effective coordinate systems for nonlinear dynamical systems.
method Developed a new algorithm to identify more stable and accurate models from less data, leveraging the connection between HAVOK and Frenet-Serret frame.
result The sub- and super-diagonal entries of the linear model correspond to intrinsic curvatures in Frenet-Serret frame.
Proposes neuron alignment to optimize mode connectivity in neural networks.
problem Understanding and optimizing mode connectivity in deep neural networks.
method Introduces neuron alignment to approximate optimal weight permutations and improve mode connectivity.
result Neuron alignment significantly alleviates robust loss barriers and improves model robustness and accuracy.
Periodic surfaces have a limited number of bending modes, equal to their membrane modes.
problem Understanding the limitations of bending modes in periodic surfaces.
method Analyzing deformation modes of periodic, piecewise smooth, simply connected surfaces.
result Effective membrane modes and bending modes are orthogonal, limiting the total number of modes to 3.
Revisits zero modes of Dirac operator on Eguchi-Hanson space.
problem Determining zero modes of the Dirac operator on Eguchi-Hanson space.
method Uses spin-c spinors and formalism of differential forms to simplify calculations. result Reproduces known normalisable zero modes of the twisted Eguchi-Hanson Dirac operator.
Proves analyticity of quasinormal modes in Kerr and Kerr-de Sitter spacetimes.
problem Analyticity of quasinormal modes in extreme Kerr and Kerr-de Sitter spacetimes.
method Observation of stable radial point source/sink structure in bicharacteristic flow; recent microlocal analysis result by Galkowski and Zworski.
result Quasinormal modes are real analytic in subextremal Kerr and Kerr-de Sitter spacetimes.
Mode clustering is a nonparametric method for clustering that defines clusters using the basins of attraction of a density estimator's modes. We provide several enhancements to mode clustering: (i) a soft variant of cluster assignment, (ii) a measure of connectivity between clusters, (iii) a technique for choosing the …
Paper discovers simplicial complexes connecting trained models for improved ensembling.
problem Improving robustness and accuracy of deep learning ensembles.
method Identifies mode-connecting simplicial complexes on loss surfaces.
result Efficiently builds simplicial complexes for ensembling, outperforming independent ensembles.
Mode connectivity is a surprising phenomenon in the loss landscape of deep nets. Optima -- at least those discovered by gradient-based optimization -- turn out to be connected by simple paths on which the loss function is almost constant. Often, these paths can be chosen to be piece-wise linear, with as few as two segm…
New method uses random projections to estimate densities and modes efficiently.
problem Estimating densities and modes from sparse representations.
method Expand-and-sparsify representations followed by linear function and mode recovery algorithms.
result Optimal rates for density and mode estimation achieved.
Empirical study shows removing neural parameter symmetries impacts model performance.
problem Understanding the impact of neural parameter symmetries on model performance.
method Developed two methods to reduce parameter space symmetries in neural networks.
result Removing parameter symmetries can lead to faster and more effective Bayesian neural network training.
LoRA-Curve connects independent LoRA optima through continuous low-loss valleys, improving Bayesian model averaging.
problem Challenges in estimating epistemic uncertainty in LoRA-based Bayesian inference.
method Introduces LoRA-Curve, a segmented Bézier curve parameterization in the LoRA space, with free and anchored configurations.
result Empirically shows that connecting independent LoRA optima through continuous low-loss valleys improves mutual information of the predictive distribution.
The convergence rate and final performance of common deep learning models have significantly benefited from heuristics such as learning rate schedules, knowledge distillation, skip connections, and normalization layers. In the absence of theoretical underpinnings, controlled experiments aimed at explaining these strate…
Study identifies failure modes of machine learning models in out-of-distribution settings.
problem Machine learning models fail to generalize well to new, unseen data.
method Theoretical study of gradient-descent-trained linear classifiers on easy-to-learn tasks, followed by experiments on modern neural networks.
result Two failure modes of spurious correlations are uncovered: geometric and statistical.
We identify and study two common failure modes for early training in deep ReLU nets. For each we give a rigorous proof of when it occurs and how to avoid it, for fully connected and residual architectures. The first failure mode, exploding/vanishing mean activation length, can be avoided by initializing weights from a …
Study on stability of 3D sessile drops, identifying degenerate kernel.
problem Linear stability of three-dimensional sessile drops with a free contact line.
method Derived constrained second variation, formulated Jacobi problem, combined geometric and Fourier analysis.
result Kernel of the constrained Jacobi operator is exactly the space of horizontal translations under pressure-volume nondegeneracy.
Optimized DMD for fast atmospheric chemistry forecasting.
problem Forecasting global atmospheric chemistry dynamics efficiently.
method Optimized Dynamic Mode Decomposition (DMD) for reduced order modeling.
result Significant improvement in computational speed and interpretability.
AutoLL uses neural networks to automatically reorder graph nodes for linear layouts.
problem Finding optimal node order in adjacency matrices without predefined features.
method Developed AutoLL-D and AutoLL-U neural network models for one-mode reordering of directed and undirected graphs.
result Demonstrated effectiveness of AutoLL through qualitative and quantitative evaluations.
Proposes MVGPR for spatiotemporal data modal analysis.
problem Sparse and irregularly sampled data in complex flows.
method Multivariate Gaussian process regression (MVGPR) with kernel design.
result MVGPR outperforms DMD and SPOD in modal analysis of sparse and irregular data.
pLSTM tackles long-range language modeling and computer vision tasks with parallelizable linear source transition mark networks.
problem Challenges of existing recurrent architectures in handling sequences and multi-dimensional data.
method Introduces pLSTM, a parallelizable linear source transition mark network for linear graphs and DAGs, addressing vanishing/exploding activation/gradient issues.
result pLSTM outperforms Transformers in long-range tasks like arrow-pointing extrapolation and image size extrapolation.
Dynamic Mode Decomposition (DMD) yields a linear, approximate model of a system's dynamics that is built from data. We seek to reduce the order of this model by identifying a reduced set of modes that best fit the output. We adopt a model selection algorithm from statistics and machine learning known as Least Angle Reg…
Multimodal clustering is an unsupervised technique for mining interesting patterns in n-adic binary relations or n-mode networks. Among different types of such generalized patterns one can find biclusters and formal concepts (maximal bicliques) for 2-mode case, triclusters and triconcepts for 3-mode case, closed $n…
Many complex dynamical phenomena can be effectively modeled by a system that switches among a set of conditionally linear dynamical modes. We consider two such models: the switching linear dynamical system (SLDS) and the switching vector autoregressive (VAR) process. Our Bayesian nonparametric approach utilizes a hiera…
Neurons predict future scalar inputs by learning top modes of lag vectors.
problem Predicting future scalar inputs with physiological delays.
method Normal Mode Decomposition to extract independently evolving modes.
result Temporal filters of neurons correspond to left eigenvectors of a generalized eigenvalue problem.
New method decomposes KL error using refined information and mode interactions.
problem Learning probability distributions over discrete variables with higher-order interactions.
method Using information geometry, refined mode interactions, and a novel Monte-Carlo sampling technique.
result Complete decomposition of KL error and efficient data use.
We study whether a neural network optimizes to the same, linearly connected minimum under different samples of SGD noise (e.g., random data order and augmentation). We find that standard vision models become stable to SGD noise in this way early in training. From then on, the outcome of optimization is determined to a …
Finsler geometry connects to higher-spin fields and curved-space Fronsdal equations.
problem Eliminating non-transverse modes in Finsler dynamics for higher spins.
method Parameterizing Finsler geometry in terms of symmetric tensors, analyzing linear and nonlinear terms, and examining gauge transformations.
result Finsler dynamics leads to the curved-space Fronsdal equation for all spins, plus a Stueckelberg-like coupling.
Generative adversarial networks (GANs) are innovative techniques for learning generative models of complex data distributions from samples. Despite remarkable recent improvements in generating realistic images, one of their major shortcomings is the fact that in practice, they tend to produce samples with little divers…
Many generative models have to combat missing modes. The conventional wisdom to this end is by reducing through training a statistical distance (such as f-divergence) between the generated distribution and provided data distribution. But this is more of a heuristic than a guarantee. The statistical distanc…
New method makes neural networks transparent, revealing learning modes.
problem Lack of interpretability in neural networks.
method Weight pathway analysis (WPA) to decompose neural networks into subnetworks.
result Neural networks store and utilize information holographically, with linear and nonlinear learning modes.
Study examines deformations of Kerr-(A)dS near horizon geometry.
problem Analyzing deformations of Kerr-(A)dS near horizon geometry.
method Two-part proof: elimination of Fourier modes and analyticity argument.
result No odd Fourier modes found for linear perturbations.
Enhances forecasting of complex systems using FKMD.
problem Forecasting high-dimensional dynamical systems with unknown features.
method Featurized Koopman Mode Decomposition (FKMD) using delay embedding and learned Mahalanobis distance.
result Improves prediction accuracy for various complex systems.
Mode connectivity is a recently introduced frame- work that empirically establishes the connected- ness of minima by finding a high accuracy curve between two independently trained models. To investigate the limits of this setup, we examine the efficacy of this technique in extreme cases where the input models are trai…
Improves conditions for mode connectivity in deep neural networks.
problem Ensuring mode connectivity in deep neural networks under practical conditions.
method Exploiting feature quality and mild over-parameterization conditions.
result Connectivity of solutions found by stochastic gradient descent confirmed under new conditions.
Proves wave equation solutions in Kerr-de Sitter spacetime have specific asymptotic expansions.
problem Analyzing solutions to wave equations in Kerr-de Sitter spacetime.
method Developed a Fredholm setup for quasinormal modes and analyzed trapping of lightlike geodesics.
result Proves asymptotic expansions of wave equation solutions up to a decay order.
Study on catenoid stability using asymmetric potentials.
problem Stability of catenoids and related potentials.
method Exact solvable one-dimensional Schrödinger operator with asymmetric Darboux-Pöschl-Teller potential.
result Explicit construction of the unstable mode of the inner catenoid.
Reduced modeling in high-dimensional reproducing kernel Hilbert spaces offers the opportunity to approximate efficiently non-linear dynamics. In this work, we devise an algorithm based on low rank constraint optimization and kernel-based computation that generalizes a recent approach called "kernel-based dynamic mode d…
The Softmax function on top of a final linear layer is the de facto method to output probability distributions in neural networks. In many applications such as language models or text generation, this model has to produce distributions over large output vocabularies. Recently, this has been shown to have limited repres…
The loss functions of deep neural networks are complex and their geometric properties are not well understood. We show that the optima of these complex loss functions are in fact connected by simple curves over which training and test accuracy are nearly constant. We introduce a training procedure to discover these hig…
This paper characterizes VAE training pathologies and their effects on tasks.
problem Characterizing VAE training pathologies and their impact on downstream tasks.
method Concretely characterizing conditions for VAE training pathologies and their connection to specific downstream tasks.
result Connects VAE training pathologies to specific downstream tasks like learning compressed and disentangled representations, adversarial robustness, and semi-supervised learning.