New theory explains signal propagation in normalization-free transformers.
problem Understanding signal propagation in normalization-free transformers.
method Deriving recurrence relations for activation statistics and APJNs across layers.
result Transformers with elementwise tanh-like nonlinearities exhibit subcritical signal propagation.
New method trains deep ResNets without normalization, achieving state-of-the-art performance.
problem Training deep ResNets without normalization layers leads to instability and lower accuracy.
method Adaptive gradient clipping and Normalizer-Free ResNets design.
result Normalizer-Free ResNets achieve 86.5% top-1 accuracy on ImageNet, matching EfficientNet-B7.
We use the reflection group trick to glue manifolds with corners that are Borel-Serre compactifications of locally symmetric spaces of noncompact type and obtain aspherical manifolds. We call these \emph{piecewise locally symmetric} manifolds. This class of spaces provide new examples of aspherical manifolds whose fund…
We define \emph{piecewise rank 1} manifolds, which are aspherical manifolds that generally do not admit a nonpositively curved metric but can be decomposed into pieces that are diffeomorphic to finite volume, irreducible, locally symmetric, nonpositively curved manifolds with π1-injective cusps. We prove smooth (sel…
A new aggregation strategy improves GNN performance and learning dynamics.
problem Improving expressivity and learning dynamics of GNNs.
method Proposes a variance-preserving aggregation function (VPA) for GNNs.
result VPA leads to increased predictive performance and improved learning dynamics.
Paper develops novel privacy mechanism for Riemannian manifold data using geometric analysis and heat diffusion.
problem Privacy-preserving estimation of generalized Frechet mean on Riemannian manifolds.
method Characterizes Renyi divergence via Harnack inequalities, introduces mechanisms based on heat diffusion and Langevin process.
result Proposes mechanisms for nonnegative and general Riemannian manifolds with detailed utility analyses.
Study classifies mapping class groups with hyperbolic actions on infinite-type surfaces.
problem Classifying mapping class groups with hyperbolic actions on infinite-type surfaces.
method Analyzing curve graphs and their translates to construct hyperbolic spaces.
result Classification of mapping class groups with hyperbolic actions.
Residual networks' depth is mathematically equivalent to expanding an implicit ensemble size.
problem Understanding why deep residual networks are effective.
method Formal analysis of residual networks as ensembles of shallow models.
result Increasing network depth is equivalent to expanding the size of an implicit ensemble, revealing a hierarchical structure.
SRMC framework reduces Monte Carlo variance by history-based sampling in high-dimensional spaces.
problem Efficient sampling in high-dimensional discrete or continuous state spaces.
method Score-Repellent Monte Carlo (SRMC) framework that summarizes history through running average of score evaluations.
result Improves estimator variance and mode coverage with constant memory usage.