Unified theory explains two failure modes of deep transformers and provides initialisation guidelines.
problem Two failure modes (rank collapse and entropy collapse) of self-attention layers in deep transformers.
method Analytical theory of signal propagation through deep transformers, using the Random Energy Model analogy.
result Simple algorithm to compute trainability diagrams for correct initialisation hyper-parameters.
Paper proposes ManiF-SMC for effective approximate machine unlearning.
problem Limited unlearning effectiveness and potential to undermine original learning objectives.
method Reformulates approximate unlearning as pushing erased samples towards semantic neighbors in retained data, using a margin-based triplet loss.
result Achieves unlearning effectiveness comparable to state-of-the-art methods while operating purely in representation space.
Unified framework for certifying LLM reliability without extra supervision.
problem Improving reliability of large language models without additional supervision.
method Unified framework using majority voting and Martingale Majority Certificate (MMC).
result Certifiable inference in LLMs with statistical guarantees and adaptive stopping rules.
Geodesics connect model modes in neural network loss landscapes.
problem Connecting modes in neural network loss landscapes.
method Reframed mode connectivity in Information Geometry, hypothesized geodesics as mode-connecting paths, proposed algorithm to approximate geodesics.
result Geodesics achieve mode connectivity in neural networks.
This work proposes a new feature for transportation mode classification using GPS trajectories.
problem Classifying transportation modes from GPS trajectories to optimize urban mobility.
method The Ordinal Pattern Transition Graph and its self-transition probability are used for classification.
result The proposed feature outperforms existing methods in transportation mode classification.
Optimal self-distillation improves generative models' velocity risk and mode recovery.
problem Improving generative models' velocity risk and mode recovery.
method Proved optimal self-distillation for rectified flow via linear probing, derived mixing coefficient, and provided validation tuning.
result Optimal self-distillation improves velocity risk and mode recovery.
Proposes using mode connectivity to improve adversarial robustness of neural networks.
problem Improving adversarial robustness of deep neural networks.
method Employing mode connectivity in loss landscapes to study adversarial robustness and propose methods for improvement.
result Path connection learned using limited bonafide data can effectively mitigate adversarial effects while maintaining original accuracy.
Simplified non-contrastive learning avoids representation collapse.
problem Training failure modes in self-supervised learning.
method Hyperdimensional computing and inductive bias.
result The approach avoids representation collapses.
We compute the noncommutative de Rham cohomology for the finite-dimensional q-deformed coordinate ring Cq[SL2] at odd roots of unity and with its standard 4-dimensional differential structure. We find that H1 and H3 have three additional modes beyond the generic q-case where they are 1-dimensional, while $H…
Proves stability of gravitational instantons, proving operator positivity.
problem Stability of gravitational instantons.
method Riemannian analog of black hole mode stability for Hermitian, non-self-dual gravitational instantons.
result Teukolsky equation is a positive definite operator on Hermitian, non-self-dual gravitational instantons.
Neural networks can learn optimal auction mechanisms and satisfy mode connectivity.
problem Optimal auction design in complex settings.
method Generalized RochetNet and affine maximizer auctions.
result Neural networks (RochetNet and generalized version) satisfy mode connectivity.
This study examines how fashion consumption affects self-confidence and buying behavior in Iranian consumers.
problem Understanding the role of self-confidence in fashion buying behavior.
method A questionnaire was used to collect data from 400 consumers in Tehran's clothing market. Structural equations and factor analysis were employed to test the model.
result Interest in fashion, personal taste, utilitarianism, and new products positively impact self-confidence, and self-confidence positively impacts fashion buying behavior.
Proposes neuron alignment to optimize mode connectivity in neural networks.
problem Understanding and optimizing mode connectivity in deep neural networks.
method Introduces neuron alignment to approximate optimal weight permutations and improve mode connectivity.
result Neuron alignment significantly alleviates robust loss barriers and improves model robustness and accuracy.
Self-attention networks localize when eigenspectrum variance is small.
problem Self-attention mechanisms can lead to rank and entropy collapses, reducing model expressivity and trainability.
method Characterized attention localization using query-key eigenspectrum variance.
result Small eigenspectrum variance prevents both rank and entropy collapses, improving model performance.
Periodic surfaces have a limited number of bending modes, equal to their membrane modes.
problem Understanding the limitations of bending modes in periodic surfaces.
method Analyzing deformation modes of periodic, piecewise smooth, simply connected surfaces.
result Effective membrane modes and bending modes are orthogonal, limiting the total number of modes to 3.
Blend-ASC improves self-consistency efficiency by dynamically allocating samples, reducing costs.
problem Efficiently applying self-consistency to large datasets is computationally expensive.
method Blend-ASC dynamically allocates samples during inference, improving efficiency.
result Blend-ASC reduces sample usage by 6.8x on average compared to vanilla self-consistency.
Revisits zero modes of Dirac operator on Eguchi-Hanson space.
problem Determining zero modes of the Dirac operator on Eguchi-Hanson space.
method Uses spin-c spinors and formalism of differential forms to simplify calculations. result Reproduces known normalisable zero modes of the twisted Eguchi-Hanson Dirac operator.
Mode clustering is a nonparametric method for clustering that defines clusters using the basins of attraction of a density estimator's modes. We provide several enhancements to mode clustering: (i) a soft variant of cluster assignment, (ii) a measure of connectivity between clusters, (iii) a technique for choosing the …
Paper discovers simplicial complexes connecting trained models for improved ensembling.
problem Improving robustness and accuracy of deep learning ensembles.
method Identifies mode-connecting simplicial complexes on loss surfaces.
result Efficiently builds simplicial complexes for ensembling, outperforming independent ensembles.
We consider the Yang-Mills flow on hyperbolic 3-space. The gauge connection is constructed from the frame-field and (not necessarily compatible) spin connection components. The fixed points of this flow include zero Yang-Mills curvature configurations, for which the spin connection has zero torsion and the associated R…
The paper reveals surprising star-shaped connectivity in neural networks.
problem Understanding mode connectivity in neural network landscapes.
method Fine-grained analysis of connectivity in overparameterized and finite minima cases.
result Star-shaped connectivity exists in neural network landscapes, suggesting near convexity.
We identify and study two common failure modes for early training in deep ReLU nets. For each we give a rigorous proof of when it occurs and how to avoid it, for fully connected and residual architectures. The first failure mode, exploding/vanishing mean activation length, can be avoided by initializing weights from a …
Multimodal clustering is an unsupervised technique for mining interesting patterns in n-adic binary relations or n-mode networks. Among different types of such generalized patterns one can find biclusters and formal concepts (maximal bicliques) for 2-mode case, triclusters and triconcepts for 3-mode case, closed $n…
In this paper, a unified susceptible-exposed-infected-susceptible-aware (SEIS-A) framework is proposed to combine epidemic spreading with individuals' on-line self-consultation behaviors. An epidemic spreading prediction model is established based on the SEIS-A framework. The prediction process contains two phases. In …
Twistor space constructions and actions are given for full Yang-Mills and conformal gravity using almost complex structures that are not, in general, integrable. These are used as the basis of a derivation of the twistor-string generating functionals for tree level perturbative scattering amplitudes of Yang-Mills and c…
Study on vortex sheet formation in Abelian gauge theories.
problem Understanding vortex sheet formation in Abelian gauge theories.
method Inspired by Allard's regularity theory, constructs approximate solutions and analyzes their perturbations.
result Establishes a geometric framework and regularity theory for the limiting defect set.
New method makes neural networks transparent, revealing learning modes.
problem Lack of interpretability in neural networks.
method Weight pathway analysis (WPA) to decompose neural networks into subnetworks.
result Neural networks store and utilize information holographically, with linear and nonlinear learning modes.
We construct the most general reducible connection that satisfies the self-dual Yang-Mills equations on a simply connected, open subset of flat R4. We show how all such connections lie in the orbit of the flat connection on R4 under the action of non-local symmetries of the self-dual Yang-Mills …
Generative adversarial networks (GANs) are innovative techniques for learning generative models of complex data distributions from samples. Despite remarkable recent improvements in generating realistic images, one of their major shortcomings is the fact that in practice, they tend to produce samples with little divers…
Many generative models have to combat missing modes. The conventional wisdom to this end is by reducing through training a statistical distance (such as f-divergence) between the generated distribution and provided data distribution. But this is more of a heuristic than a guarantee. The statistical distanc…
Spectral flow connects manifold geometry to rigidity criteria.
problem Tackling rigidity of simply-connected closed manifolds.
method Spectral deformation flow and invariant-based approach.
result Spherical profile is the unique manifold-compatible asymptotic realization.
Mode connectivity is a recently introduced frame- work that empirically establishes the connected- ness of minima by finding a high accuracy curve between two independently trained models. To investigate the limits of this setup, we examine the efficacy of this technique in extreme cases where the input models are trai…
Improves conditions for mode connectivity in deep neural networks.
problem Ensuring mode connectivity in deep neural networks under practical conditions.
method Exploiting feature quality and mild over-parameterization conditions.
result Connectivity of solutions found by stochastic gradient descent confirmed under new conditions.
This work introduces a novel system for the generation of images that contain multiple classes of objects. Recent work in Generative Adversarial Networks have produced high quality images, but many focus on generating images of a single object or set of objects. Our system addresses the task of image generation conditi…
Study on catenoid stability using asymmetric potentials.
problem Stability of catenoids and related potentials.
method Exact solvable one-dimensional Schrödinger operator with asymmetric Darboux-Pöschl-Teller potential.
result Explicit construction of the unstable mode of the inner catenoid.
This paper is devoted to the study of the singularity phenomenon of timelike extremal hypersurfaces in Minkowski spacetime R1+3. We find that there are two explicit lightlike self-similar solutions to a graph representation of timelike extremal hypersurfaces in Minkowski spacetime R1+3, the …
Improved motion prediction for self-driving cars using trajectory sets and auxiliary losses.
problem Accurately predicting future vehicle motion for self-driving cars.
method Classification over trajectory sets with an auxiliary loss for off-road predictions and spatial-temporal relationships.
result Significant improvement in motion prediction performance on small datasets using map information.
Model stitching compares neural representations, revealing insights not captured by CKA.
problem Understanding internal neural representations.
method Model stitching connects neural network layers to study representations.
result Good networks trained differently can be stitched without performance drop.
Under a vanishing hypothesis, Donaldson and Friedman proved that the connected sum of two self-dual Riemannian 4-Manifolds is again self-dual. Here we prove that the same result can be extended over to the positive scalar curvature case.
The loss functions of deep neural networks are complex and their geometric properties are not well understood. We show that the optima of these complex loss functions are in fact connected by simple curves over which training and test accuracy are nearly constant. We introduce a training procedure to discover these hig…
Constructs flow lines connecting unstable to stable self-expanders.
problem Existence of monotone Morse flow lines for expander functionals.
method Constructs a singular Morse flow line connecting unstable to stable self-expanders.
result Constructs a monotone flow line with a small singular set.
This paper characterizes VAE training pathologies and their effects on tasks.
problem Characterizing VAE training pathologies and their impact on downstream tasks.
method Concretely characterizing conditions for VAE training pathologies and their connection to specific downstream tasks.
result Connects VAE training pathologies to specific downstream tasks like learning compressed and disentangled representations, adversarial robustness, and semi-supervised learning.
This work builds the connection between the regularity theory of optimal transportation map, Monge-Ampère equation and GANs, which gives a theoretic understanding of the major drawbacks of GANs: convergence difficulty and mode collapse. According to the regularity theory of Monge-Ampère equation, if the support of the …
A new formalism simplifies SO(3) Yang-Mills theory connections.
problem Simplifying connections in SO(3) Yang-Mills theory.
method Pointwise polar decomposition and field h determination. result Equivalence of Yang-Mills equation to Φg(h)=0. Models for self-equivalences and diffeomorphisms of manifolds.
problem Classifying spaces of self-equivalences and diffeomorphisms of manifolds.
method Construct rational models using equivariant algebraic methods.
result Formula for rational cohomology of classifying spaces.
Krein's formula for conic Laplacians on compact Riemann surfaces
problem Establishing Krein's formula for self-adjoint extensions of conic Laplacians on compact Riemann surfaces
method Using finite-dimensional symplectic space of critical asymptotic boundary data
result Deriving a trace identity for the resolvent difference and proving a comparison formula for the positive-spectrum zeta determinants
In recent years, deep learning methods have outperformed other methods in image recognition. This has fostered imagination of potential application of deep learning technology including safety relevant applications like the interpretation of medical images or autonomous driving. The passage from assistance of a human d…
Skeleton is a new notion designed for constructing space-filling curves of self-similar sets. It is shown in [Dai, Rao and Zhang, Space-filling curves of self-similar sets (II): Edge-to-trail substitution rule,https://doi.org/10.1088/1361-6544/ab1275] that for a connected self-similar set, space-filling curves can be c…