This paper interprets Dropout as creating an equiangular tight frame in autoencoders.
problem Understanding the success of Dropout in deep learning.
method Connecting Dropout to analog channel coding and equiangular tight frames (ETF).
result Optimizing autoencoders with dropout leads to an equiangular tight frame structure.
Simply connected spaces of tight frames identified.
problem Understanding the connectivity of spaces of tight frames.
method Viewing tight frames as elements of Stiefel manifolds and identifying simply connected spaces.
result Spaces of tight frames, including finite unit-norm tight frames, are simply connected.
Gradient descent constructs tight fusion frames.
problem Constructing tight fusion frames from prescribed subspaces.
method Gradient descent and symplectic geometry.
result Gradient descent can be used to construct tight fusion frames.
Study equiangular surfaces in 3D, extending plane spirals.
problem Understanding 3D surfaces with constant normal-vector angles.
method Investigates three-dimensional extensions of equiangular spirals.
result Identifies self-similar structures in sea shell geometry.
Sparse coding in learned dictionaries has been established as a successful approach for signal denoising, source separation and solving inverse problems in general. A dictionary learning method adapts an initial dictionary to a particular signal class by iteratively computing an approximate factorization of a training …
The paper uncovers symmetries in large language models through layer-peeled optimization.
problem Understanding geometric structure in large language model weights and context embeddings.
method Constrained layer-peeled optimization program to analyze symmetries in next-token distributions.
result Symmetries in target next-token distributions are transferred to optimal model weights and context embeddings.
We analyze deep neural networks using convex duality to reveal hidden layer structures.
problem Understanding the structure of deep neural networks.
method Introducing a convex analytic framework to characterize hidden layer weights.
result Optimal hidden layer weights align with previous layers via duality.
An equiangular hyperbolic Coxeter polyhedron is a hyperbolic polyhedron where all dihedral angles are equal to π/n for some fixed integer n at least 2. It is a consequence of Andreev's theorem that either n=3 and the polyhedron has all ideal vertices or that n=2. Volume estimates are given for all equiangular hyperboli…
Deep nets exhibit 'Neural Collapse' during training's final phase, simplifying decision-making.
problem Understanding and optimizing deep learning training phases.
method Direct measurements on three deepnet architectures across seven datasets.
result Deep nets exhibit 'Neural Collapse' during training's final phase, simplifying decision-making.
Frames for Rn can be thought of as redundant or linearly dependent coordinate systems, and have important applications in such areas as signal processing, data compression, and sampling theory. The word "frame" has a different meaning in the context of differential geometry and topology. A moving frame for the tang…
Language models allocate information storage, not collapsing into uniform representations.
problem Incomplete neural collapse in language model representations.
method Analyzing variance and information sharing across 14 models, proving an information floor.
result Within-class variance is allocated information storage, not collapsed into uniform representations.
Redundancy helps speed up slow nodes in distributed learning.
problem Slow nodes (stragglers) bottleneck distributed optimization and learning performance.
method Encode data with redundancy, dynamically exclude stragglers, and compensate losses.
result Optimization algorithms converge to solutions even with straggling nodes.
A new loss function HUG decouples and generalizes neural collapse.
problem Neural collapse limits in deep learning models.
method Hyperspherical uniformity gap (HUG) as a unified framework.
result HUG decouples and generalizes neural collapse, improving model flexibility and robustness.
This paper studies how to compress neural networks while maintaining accuracy.
problem Compressing a two-layer neural network with fewer nodes without losing accuracy.
method Using tools from high-dimensional probability, the authors minimize the L_2 loss between the target and compressed networks.
result The error rate of the approximation is shown as a function of input dimension and network size in the mean-field limit.
This work justifies neural collapse under MSE loss and analyzes the optimization landscape.
problem Understanding neural collapse in deep neural networks under MSE loss.
method Global landscape analysis of vanilla nonconvex MSE loss.
result The only global minimizers are neural collapse solutions.
Graphs with maximum degree Δ have at most O(1) equiangular lines for λ < 3/sqrt(2).
problem Finding the maximum number of equiangular lines in graphs with a given maximum degree.
method Using eigenfunctions and nodal domains to estimate the multiplicity of eigenvalues.
result The maximum multiplicity of λ as the second largest eigenvalue is O(1) for graphs with maximum degree Δ and cyclomatic number.
We study the configuration space of equilateral and equiangular spatial hexagons for any bond angle by giving explicit expressions of all the possible shapes. We show that the chair configuration is isolated, whereas the boat configuration allows one-dimensional deformations which form a circle in the configuration spa…
New model explains neural collapse and limits on minority classes in imbalanced datasets.
problem Understanding and predicting performance limits of deep learning models on imbalanced datasets.
method Layer-Peeled Model, a nonconvex optimization program isolating top layers and applying constraints.
result Reveals a new phenomenon called Minority Collapse that limits deep learning models on minority classes.
Deep linear networks exhibit collapsing features and classifiers across datasets.
problem Understanding the collapse of features and classifiers in deep linear networks.
method Theoretical and empirical analysis of deep linear networks with MSE and CE losses.
result Deep linear networks exhibit NC properties, collapsing features and classifiers to orthogonal vectors.
New method denoises graph signals using wavelets, scalable for large graphs.
problem Denoising graph signals with overcomplete tight frames and correlated noise.
method Data-driven wavelet tight frame, Stein's unbiased risk estimate, Chebyshev-Jackson polynomial approximations, Monte-Carlo strategy.
result Method scales to large graphs and finds applications in differential privacy.
This paper extends neural collapse to class-imbalanced datasets using an unconstrained ReLU feature model.
problem Understanding neural collapse in class-imbalanced datasets with cross-entropy loss.
method Generalized neural collapse to class-imbalanced settings using an unconstrained ReLU feature model.
result Class-means converge to orthogonal vectors with different lengths, and classifier weights align to these vectors.
Quantizes neural networks using frame theory for improved accuracy.
problem Improving neural network efficiency and accuracy through quantization.
method Sigma-Delta (ΣΔ) quantization with finite unit-norm tight frames. result Error bound between original and quantized neural networks derived.
Quantifies how geodesic planes isolate in hyperbolic 3-manifolds.
problem Understanding isolation properties of geodesic planes in hyperbolic 3-manifolds.
method Quantitative estimates of geodesic planes in frame bundles, using tight areas and densities.
result Polynomial estimates of isolation properties with degree given by modified critical exponents.
Pipeline combines ETF preprocessing with tabular model for cross-modal inference.
problem Transferability of tabular models across different modalities.
method Fixed comparison object, ETF preprocessing, in-context inference.
result Pipeline is broadly competitive, runs faster, and produces well-calibrated probabilities.
We show that an oriented elliptic 3-manifold admits a universally tight positive contact structure iff the corresponding group of deck transformations on S3 preserves a standard contact structure pointwise. We also relate univerally tight contact structures on 3-manifolds covered by S3 to the exceptional isomorph…
We analyze neural collapse in neural networks, showing that features collapse to vertices of a Simplex ETF.
problem Understanding and optimizing the features learned in the last layer of neural networks during training.
method Simplified unconstrained feature model, studying the global optimization landscape of cross-entropy loss with weight decay.
result The global minimizers of the loss are Simplex ETFs, and other critical points are strict saddles with negative curvature.
Deep nets trained with MSE loss exhibit Neural Collapse, collapsing features and classifiers to class means.
problem Understanding Neural Collapse in MSE-trained deep nets.
method Developed a new MSE loss decomposition and introduced the central path concept.
result Exact dynamics of Neural Collapse along the central path can be predicted.
We study pairs of curves with Poncelet's porism properties and compute their vertex curves.
problem Understanding pairs of curves with Poncelet's porism properties.
method Developed formulas to compute vertex curves for given envelope curves and vice versa, for all sufficiently regular pairs of Poncelet curves.
result Formulas produce all possible sufficiently regular pairs of Poncelet curves, including sets of curves analogous to pencils of conic sections.
New wavelet frames constructed from reproducing kernels for continuous and discrete domains.
problem Generating wavelet frames on non-Euclidean structures.
method Spectral filtering of integral operators associated with reproducing kernels.
result Discrete frames as Monte Carlo estimates of continuous frames, with finite-sample rates derived.
The paper classifies tight contact structures on Seifert fiber spaces.
problem Classifying tight contact structures on Seifert fiber spaces.
method Using Legendrian surgery and convex surface theory.
result Tight contact structures on certain Seifert fiber spaces are classified.
This paper extends neural collapse to imbalanced data under cross-entropy loss.
problem Analyzing neural collapse in deep networks with imbalanced data.
method Using the unconstrained feature model and cross-entropy loss, the paper studies neural collapse in imbalanced datasets.
result Feature vectors within the same class collapse to a single mean vector, but angles between them depend on sample size.
It is well-known that a knot in a contact manifold (M,C) transverse to a trivialized contact structure possesses the natural framing given by the first of the trivialization vectors along the knot. If the Euler class eC∈H2(M) of C is nonzero, then C is nontrvivializable and the natural framing of transvers…
Six quaternionic lines with optimal angles found in 2D quaternion space.
problem Finding optimal configurations of quaternionic lines in 2D space.
method Simple presentation of lines as orbit of a reflection group, finding other optimal designs.
result Optimal spherical designs of 10, 15, and 20 lines in quaternion space.
FPCA optimizes fairness in target vectors' span.
problem Fairness in principal component analysis for multiple target vectors.
method Non-concave maximization of worst projected target norm using sub-gradient descent.
result Optimization landscape is benign with globally optimal local minima.
This work explains neural collapse in shallow neural networks and its impact on generalization.
problem Understanding neural collapse in shallow neural networks and its effect on generalization.
method Analysis of two and three-layer ReLU neural networks, focusing on data dimension, sample size, and signal-to-noise ratio.
result Neural collapse occurs in shallow ReLU networks under certain conditions related to data properties and network architecture.
Parseval networks improve deep nets' robustness to adversarial examples.
problem Improving deep neural networks' robustness to adversarial attacks.
method Constraining the Lipschitz constant and maintaining Parseval tight frames in weight matrices.
result Parseval networks maintain accuracy and robustness to adversarial examples compared to vanilla networks.
New deep learning methods improve CT image quality from few projections.
problem Sparse-view CT images suffer from streaking artifacts due to limited projections.
method Inspired by deep convolutional framelets, propose new U-Net variants that satisfy the frame condition.
result New U-Net variants provide better reconstruction performance for sparse-view CT.
New hyperbolic polyhedra with π/3 angles and volumes calculated.
problem Finding new hyperbolic polyhedra with specific dihedral angles.
method Constructed a new sequence of hyperbolic polyhedra with π/3 angles and determined their volumes. result Volumes of some constructed polyhedra determined.
Paper tackles image reconstruction from limited data using polyhedral norms and convex regularizers.
problem Learning convex regularizers for image reconstruction from limited data.
method Imposes amplitude-equivariance, approximates functionals with polyhedral norms, identifies synthesis and analysis forms, proposes a trainable tight frame architecture.
result Proposed framework outperforms sparsity-based methods in denoising and biomedical image reconstruction.
Round surgery diagrams represent 3-manifolds in S3.
problem Representing and manipulating 3-manifolds in S3. method Introducing round surgery diagrams and defining moves to establish Kirby Calculus.
result Any 3-manifold can be obtained by a round surgery on a framed link in S3. We construct a simple topological invariant of certain 3-manifolds, including quotients of the 3-sphere by finite groups, based on the fact that the tangent bundle of an orientable 3-manifold is trivialisable. This invariant is strong enough to yield the classification of lens spaces of odd, prime order. We also use pr…
A standard convexity condition on the boundary of a symplectic manifold involves an induced positive contact form (and contact structure) on the boundary; the corresponding concavity condition involves an induced negative contact form. We present two methods of symplectically attaching 2-handles to convex boundaries of…
A twisted curve in Euclidean 3-space E^3 can be considered as a curve whose position vector can be written as linear combination of its Frenet vectors. In the present study we study the twisted curves of constant ratio in E^3 and characterize such curves in terms of their curvature functions. Further, we obtain some re…
SSTQ improves privacy-preserving vector quantization with low communication cost.
problem Achieving local differential privacy in distributed optimization with low communication cost.
method Combines overcomplete equal-norm tight frames, coordinate subsampling, and privacy-aware one-dimensional quantization.
result Achieves optimal mean squared error scaling with only ⌈log2N⌉+b bits per client. A theory of feature geometry using spectral analysis of weight matrices.
problem Current methods decompose neural network activations into sparse linear features, losing geometric structure.
method Develops a theory by analyzing the spectra of weight-derived matrices, introducing the frame operator.
result Features collapse onto single eigenspaces, organizing into tight frames, and admit discrete classification.
New method interprets ranked data on permutahedron graph.
problem Interpreting and exploiting structure in ranked data sets.
method Combining combinatorial representation theory and signal processing on graphs.
result Developed scalable transform method using Parseval frames.
New set class preserves Fourier series terms for planar ovals, leading to isoperimetric inequalities.
problem Investigate geometric properties of kth Order Preserving Sets and ovals. method Introduce and analyze kth Order Preserving Sets and Midpoint Sets; study geometric properties and isoperimetric inequalities. result Established an isoperimetric-type inequality relating perimeter and area of ovals and their associated sets.
In this paper, we study the global behaviour of contact structures on oriented manifolds V which are circle bundles over a closed orientable surface S of genus g>0. We establish in particular contact analogs of a number of classical results about foliations due to Milnor, Wood, Thurston, Matsumoto, and Ghys. In Section…