GD iterates for non-homogeneous deep nets increase margin and converge in direction.
problem Understanding implicit bias in non-homogeneous deep networks.
method Characterization of GD iterates' properties starting from small empirical risk.
result GD iterates converge in direction despite diverging norms, satisfying KKT conditions.
This work analyzes the maximum-margin bias in quasi-homogeneous neural networks.
problem Analyzing the maximum-margin bias in quasi-homogeneous neural networks.
method Geometric analysis of gradient dynamics for quasi-homogeneous models.
result Gradient flow implicitly favors a subset of parameters, leading to asymmetric norm minimization.
Neural network approach simplifies multiscale problem homogenization.
problem Homogenizing multiscale problems with varying microscale structures.
method Derivative-free neural network with Brownian walkers.
result Neural network method is computationally efficient and robust.
The paper analyzes neural network dynamics after weights escape the origin.
problem Understanding gradient flow dynamics of neural networks after the origin.
method Analyzes gradient flow of homogeneous neural networks with locally Lipschitz gradients.
result Characterizes the first saddle point encountered after escaping the origin.
The paper generalizes equivariant neural networks on homogeneous spaces to the non-linear setting.
problem Equivariant neural networks on homogeneous spaces.
method Deriving generalized steerability constraints for non-linear equivariant layers.
result The universality of the derived construction for non-linear equivariant layers.
We embed KKT points in neural networks of different sizes.
problem Classifying data using homogeneous neural networks.
method Introducing KKT point embedding principle and proving it for different network types.
result KKT points of a smaller network can be mapped to those of a larger network via linear transformations.
GCNNs on homogeneous spaces use vector bundles and Hilbert spaces.
problem Learning data on homogeneous spaces with global symmetry.
method Analysis of G-equivariant convolutional layers on homogeneous G/K spaces, using vector bundles and reproducing kernel Hilbert spaces. result A precise criterion for expressing G-equivariant layers as convolutional layers, leading to stronger results for some groups. We develop a convex relaxation method for analyzing neural network generalization.
problem Analyzing the generalization of parallel positively homogeneous networks.
method Linking non-convex ERM to a convex optimization problem over prediction functions.
result Achieved generalization bounds with almost linear sample complexity in network width.
Large GD stepsizes improve margins and speed up training for non-homogeneous networks.
problem Training efficiency and margin improvement in non-homogeneous two-layer networks.
method Investigation of two distinct phases in GD training, showing margin growth and empirical risk decrease.
result Large GD stepsizes lead to faster convergence and improved margins in non-homogeneous networks.
A new method detects communities in multi-relational networks.
problem Detect communities in heterogeneous multi-relational networks.
method Message passing based algorithm on a hierarchical structure of homogeneous networks.
result Effectiveness of the proposed method confirmed on synthetic and real-world networks.
Neural networks can approximate positive homogeneous functions, especially with multiple hidden layers.
problem Approximating positive homogeneous functions with neural networks.
method Using scale-invariant ReLU networks with multiple hidden layers.
result Approximation of positive homogeneous functions is possible with neural networks, especially with two hidden layers.
Early training of deep neural networks leads to small, directionally converging weights.
problem Training dynamics of deep homogeneous neural networks with small initializations.
method Gradient flow analysis and study of KKT points for neural correlation function.
result Weights converge in direction to KKT points during early training stages.
New insights into continual learning for deep models, showing convergence issues but local linear solutions.
problem Challenges in continual learning for homogeneous deep models.
method Sequential projections onto task margin sets, leveraging nonconvex projection theory.
result Local linear convergence under certain conditions for homogeneous deep networks.
SGD converges to critical points of normalized margin in late-stage training for homogeneous neural networks.
problem Analyzing the implicit bias of SGD on homogeneous neural networks.
method Interpreting SGD dynamics as an Euler-like discretization of a conservative field flow associated with the normalized classification margin.
result Normalized SGD iterates converge to the set of critical points of the normalized margin at late-stage training.
An overwhelming number of true and false news stories are posted and shared in social networks, and users diffuse the stories based on multiple factors. Diffusion of news stories from one user to another depends not only on the stories' content and the genuineness but also on the alignment of the topical interests betw…
In this paper, we study the implicit regularization of the gradient descent algorithm in homogeneous neural networks, including fully-connected and convolutional neural networks with ReLU or LeakyReLU activations. In particular, we study the gradient descent or gradient flow (i.e., gradient descent with infinitesimal s…
The paper studies neural networks' convergence near origin and saddle points.
problem Directional convergence of neural networks near small initializations and saddle points.
method Gradient flow dynamics analysis of two-homogeneous neural networks.
result Neural networks' weights approximately converge in direction to KKT points for small initializations.
We study the problem of identifying different behaviors occurring in different parts of a large heterogenous network. We zoom in to the network using lenses of different sizes to capture the local structure of the network. These network signatures are then weighted to provide a set of predicted labels for every node. W…
We present a general theory of Group equivariant Convolutional Neural Networks (G-CNNs) on homogeneous spaces such as Euclidean space and the sphere. Feature maps in these networks represent fields on a homogeneous base space, and layers are equivariant maps between spaces of fields. The theory enables a systematic cla…
New regularizer improves neural network robustness and generalization.
problem Ineffective weight decay for networks with homogeneous activation functions.
method Proposes an invariant regularizer to penalize intrinsic weight norms.
result Improves generalization and adversarial robustness on various datasets.
Study shows how steepest descent algorithms' geometric margin increases during training.
problem Understanding implicit bias in steepest descent algorithms for neural networks.
method Analysis of steepest descent algorithms with infinitesimal learning rates in homogeneous neural networks.
result Limit points of training trajectories correspond to KKT points of margin-maximization problems.
GHNet improves graph learning by balancing homogeneity and heterogeneity.
problem Over-smoothing in GCN leads to similar node representations.
method GHNet uses gating units to balance homogeneity and heterogeneity in feature propagation.
result GHNet achieves larger receptive fields without over-smoothing.
AEGCN uses autoencoder constraints to improve graph node classification.
problem Node classification on graph domains with reduced information loss.
method Autoencoder-constrained graph convolutional network (AEGCN).
result Adding autoencoder constraints significantly improves graph convolutional network performance.
Study on geodesic distances on SE(3)/SO(2) in machine learning.
problem Investigating the efficiency of computationally efficient sections in selecting geodesic distances.
method Analyzing geodesic distances on reductive homogeneous spaces, proving the efficiency of minimal distance sections.
result Minimal distance sections are not always geodesic minimizers, but minimal horizontal geodesics are.
Study shows momentum-based optimizers like Muon and MomentumGD bias towards KKT points in smooth homogeneous models.
problem Understanding the implicit bias of momentum-based optimizers on smooth homogeneous models.
method Analysis of Muon, MomentumGD, Signum, and Adam optimizers under decaying learning rate schedules.
result Momentum-based optimizers approximate steepest descent trajectories and bias towards KKT points of margin maximization problems.
New strategies improve multi-agent decision-making on irregular networks.
problem Maximizing group reward in multi-agent settings with heterogeneous strategies.
method Design and analysis of heterogeneous explore-exploit strategies for multi-star networks.
result Group performance improves under heterogeneous strategies compared to homogeneous strategies.
WaveletGAN improves GANs by homogenizing noise through multi-channel wavelet filtering.
problem Current noise generation models in GANs struggle with homogenizing noise, leading to low-fidelity samples.
method Proposes a multi-channel wavelet-based filtering method to homogenize noise in GANs.
result WaveletGAN generates high-fidelity samples with the smallest FIDs on Fashion-MNIST, KMNIST, and SVHN datasets.
Develops wavelet-based neural network approximation theory.
problem Analyzing neural network approximation capabilities over various activation functions.
method Wavelet frame theory on spaces of homogeneous type, sufficient conditions for approximation, error estimates.
result Derives sufficient conditions for neural networks to approximate any functions in a given space, including non-smooth activations.
Proposes a new Bayesian score for learning network structure from related datasets.
problem Learning network structure from heterogeneous related data sets.
method Bayesian Hierarchical Dirichlet (BHD) score based on a hierarchical model.
result BHD outperforms BDeu in reconstruction accuracy and sparsity for related datasets.
It has been empirically observed that the flatness of minima obtained from training deep networks seems to correlate with better generalization. However, for deep networks with positively homogeneous activations, most measures of sharpness/flatness are not invariant to rescaling of the network parameters, corresponding…
New algorithm detects communities in weighted networks, improving on binary ones.
problem Few methods exist for detecting communities in weighted networks.
method Pseudo-likelihood approach for weighted stochastic block model.
result The method is consistent and works well for both homogeneous and heterogeneous networks.
The paper connects flatness to generalization in learning multi-index models with neural networks.
problem Understanding the generalization of non-convex neural networks using flatness measures.
method Analyzes 2-layer non-convex homogeneous neural networks and their connection to multi-index models.
result Flattest interpolators achieve small population loss and generalize well, establishing a direct link between flatness and generalization.
Classical results on the statistical complexity of linear models have commonly identified the norm of the weights ∥w∥ as a fundamental capacity measure. Generalizations of this measure to the setting of deep networks have been varied, though a frequently identified quantity is the product of weight norms of each la…
In a spatially embedded network, that is a network where nodes can be uniquely determined in a system of coordinates, links' weights might be affected by metric distances coupling every pair of nodes (dyads). In order to assess to what extent metric distances affect relationships (link's weights) in a spatially embedde…
A new algorithm trains deep neural networks by adding neurons greedily.
problem Training deep neural networks efficiently and effectively.
method Neuron Pursuit (NP) algorithm, which alternates between neuron addition and loss minimization.
result The algorithm can train deep neural networks efficiently and effectively.
Introduces new types of homogeneous spaces and their properties.
problem Defining and understanding new types of homogeneous spaces.
method Introducing and analyzing (strongly) (Θ-)discrete homogeneous spaces. result Discovers relationships between new and existing homogeneous space types.
Group equivariant neural networks simplify complex tasks with group representation theory.
problem Challenging tasks requiring input transformations like rotations.
method Group representation theory, non-commutative harmonic analysis, differential geometry.
result A neural network is group equivariant if and only if it has a convolutional structure.
Paper improves Bayesian network learning from related data sets.
problem Learning from heterogeneous data sets with different probabilistic structures.
method Mixed-effects models to pool information across related data sets.
result Mixed-effects models outperform traditional methods in accuracy.
Heavy-tailed distributions emerge in SGD's parameter evolution.
problem Understanding heavy-tailed distributions in SGD parameter evolution.
method Continuous diffusion approximation of SGD (homogenized SGD) analysis.
result Explicit upper and lower bounds on tail-index of homogenized SGD.
Attention-based GNNs can't prevent oversmoothing, leading to homogeneous node representations.
problem The issue of oversmoothing in attention-based GNNs.
method Viewed attention-based GNNs as nonlinear time-varying dynamical systems and used tools from the theory of products of inhomogeneous matrices and the joint spectral radius.
result Graph attention mechanism cannot prevent oversmoothing and loses expressive power exponentially.
Introduces homogeneity supermanifolds for studying graded structures.
problem Graded structures on supermanifolds.
method Homogeneity degrees and weight vector field.
result Proofs of homogeneous Poincaré Lemma, Frobenius Theorem, and Darboux Theorem.
Unified theory of deep neural networks with diverse activations.
problem Understanding the relationship between depth and complexity in deep neural networks.
method Developed a unified function space theory for deep networks with various activations.
result Unified theory provides meaningful complexity for deep networks with diverse activations.
Recently, it is shown that each regular homogeneous Finsler space M admits at least one homogeneous geodesic through any point o∈M. The purpose of this article is to study the existence of homogeneous geodesics on singular homogeneous (α,β)-spaces, specially, homogeneous Kropina spaces. We show that any homoge…
In this paper, we study homogeneous geodesics in homogeneous Finsler spaces. We first give a simple criterion that characterizes geodesic vectors. We show that the geodesics on a Lie group, relative to a bi-invariant Finsler metric, are the cosets of the one-parameter subgroups. The existence of infinitely many homogen…
Survival models predict component failures using neural networks and resampled data.
problem Accurately predicting component failure times for maintenance planning.
method Neural network-based survival models trained on non-independent, homogeneously sampled data.
result Random resampling during training reduces dataset size and improves efficiency.
We prove that the Penrose limit of a spacetime along a homogeneous geodesic is a homogeneous plane wave spacetime and that the Penrose limit of a reductive homogeneous spacetime along a homogeneous geodesic is a Cahen--Wallach space. We then consider several homogenous examples to show that these results are indeed sha…
Bayesian PINNs learn elliptic PDEs with near-minimax posterior contraction rate.
problem Learning elliptic PDEs with noisy data and non-homogeneous boundary conditions.
method Bayesian approach with a Hölder space prior on neural network weights.
result Posterior contracts at near-minimax rate without prior knowledge of solution smoothness.
We prove that under some purely algebraic conditions every locally homogeneous structure modelled on some homogeneous space is induced by a locally homogeneous structure modelled on a different homogeneous space.