The paper explores how sinks and diagonal patterns prevent attention oversmoothing.
problem Preventing attention oversmoothing in neural networks.
method Analyzing geometric conditions and conditions for dense vs. sparse attention, proving equivalence between sinks and hard attention switch, and comparing the costs of sinks vs. diagonal patterns.
result Sinks and diagonal patterns effectively prevent attention oversmoothing, and diagonal patterns provide a more flexible approach.
Gradient oversmoothing and expansion hinder deep GNN training, solved with normalization.
problem Gradient oversmoothing and expansion prevent deep GNN training.
method Proposed normalization method to constrain the Lipschitz bound of each layer.
result Residual GNNs with hundreds of layers can be efficiently trained with the proposed normalization.
Attention-based GNNs can't prevent oversmoothing, leading to homogeneous node representations.
problem The issue of oversmoothing in attention-based GNNs.
method Viewed attention-based GNNs as nonlinear time-varying dynamical systems and used tools from the theory of products of inhomogeneous matrices and the joint spectral radius.
result Graph attention mechanism cannot prevent oversmoothing and loses expressive power exponentially.
FoSR adds edges to graphs to prevent oversquashing and oversmoothing in GNNs.
problem Oversquashing and oversmoothing in graph neural networks (GNNs).
method First-order spectral rewiring to add edges based on spectral expansion, combined with a relational architecture.
result Our algorithm outperforms existing graph rewiring methods in graph classification tasks.
Graph neural networks suffer from oversmoothing, but adding residual connections helps.
problem Oversmoothing in deep graph neural networks where features become indistinguishable.
method Analyzed asymptotic oversmoothing rates with and without residual connections using the multiplicative ergodic theorem.
result Adding residual connections effectively mitigates or prevents oversmoothing.
ContraNorm prevents dimensional collapse in GNNs and Transformers.
problem Dimensional collapse in Graph Neural Networks and Transformers.
method Proposes ContraNorm, a novel normalization layer inspired by contrastive learning.
result Proves ContraNorm alleviates both complete and dimensional collapse under certain conditions.
The performance of graph neural nets (GNNs) is known to gradually decrease with increasing number of layers. This decay is partly attributed to oversmoothing, where repeated graph convolutions eventually make node embeddings indistinguishable. We take a closer look at two different interpretations, aiming to quantify o…
Graphs avoid oversmoothing with properly initialized weights.
problem Oversmoothing in graph convolutional networks (GCNs) limits their depth.
method Used Gaussian process equivalence to analyze GCNs, identifying a non-oversmoothing phase.
result Properly initialized GCNs can be deep and expressive.
The paper analyzes oversmoothing in GNNs and quantifies the effects of mixing and denoising.
problem Oversmoothing in Graph Neural Networks (GNNs).
method Non-asymptotic analysis of graph convolutions and effects of mixing and denoising.
result The number of layers required for oversmoothing to occur is O(logN/log(logN)) for dense graphs. New metrics reveal oversmoothing in GNNs more accurately than traditional methods.
problem Oversmoothing in graph neural networks reduces model performance.
method Rank-based metrics to measure oversmoothing in GNNs.
result Rank-based metrics consistently capture oversmoothing, while energy-based metrics often fail.
New view: Deep GCNs learn to anti-oversmooth during training.
problem Performance drop in deep GCNs due to oversmoothing.
method Interpreted GCN as MLP + graph regularization, analyzed training process.
result Deep GCNs learn to anti-oversmooth during training, not over-smooth.
Graph transformers outperform graph convolutions by preserving community information.
problem Understanding why graph transformers perform well in node-level prediction tasks.
method Analyzing the Gaussian process limits of graph transformers with infinite width and infinite heads.
result Graph transformers maintain discriminative node representations even in deep layers, preventing oversmoothing.
Scattering GCN improves graph neural networks by filtering oversmoothing.
problem Oversmoothing in GCNs limits their ability to distinguish graph nodes.
method Augmenting GCNs with geometric scattering transforms and residual convolutions.
result Scattering GCN outperforms GAT in semi-supervised node classification.
A new graph neural network tackles oversmoothing and generalization issues.
problem Oversmoothing and poor generalization for unseen graphs in graph neural networks.
method Graph Entities with Step Mixture via random walk (GESM) that considers both edge-based and node-based features.
result GESM achieves state-of-the-art or comparable performances on benchmark datasets.
A new hybrid GNN framework tackles oversmoothing in graph data.
problem Oversmoothing in graph convolutional networks limits their expressive power and generalization.
method Combines traditional GCN filters with band-pass filters defined via geometric scattering and introduces an attention framework.
result Improves expressive power and generalization of graph convolutional networks.
The paper establishes principles for initializing and designing GNNs with ReLU activations to avoid oversmoothing and correlation collapse.
problem Oversmoothing and correlation collapse in deep ReLU GNNs.
method The paper derives and validates three principles for initialization and architecture selection in finite width graph neural networks with ReLU activations.
result Correct initialization, residual aggregation operators, and residual connections significantly improve early training dynamics in deep ReLU GNNs.
GNNs may be limited by graph topology, affecting their learning outcomes.
problem Understanding how graph topology influences GNN behavior and performance.
method Investigating the interaction between local topological features and GNN message-passing schemes.
result Locally similar neighborhoods can lead to consistent node representations, affecting GNN performance.
Graph smoothing can improve learning performance by restoring lost information.
problem Graph Neural Networks (GNNs) can oversmooth, losing important information.
method Analyzing simplified linear GNNs with mean aggregation, showing benefits up to a certain point.
result Graph smoothing can restore lost information, improving both regression and classification.
Graph-Coupled Oscillator Networks (GraphCON) tackles graph-based learning problems.
problem The oversmoothing problem in Graph Neural Networks (GNNs).
method GraphCON is a novel framework based on discretizations of ODEs modeling oscillators coupled via graph adjacency.
result GraphCON mitigates the oversmoothing problem and exploding/vanishing gradients issues.
Hop Sampling improves GNNs in non-stationary environments by preventing overfitting.
problem Non-stationary environments cause concept drift, making GNNs overfit to training graphs.
method Randomly selects the number of propagation steps in GNNs to prevent overfitting.
result Improves GNNs' prediction accuracy by 7.97% and 16.93% in LINE Coupon recommender systems.
Corrected graph convolutions improve node classification on graphs.
problem Oversmoothing in graph convolutions degrades performance.
method Theoretical analysis based on CSBM, spectral analysis for k rounds of corrected graph convolutions.
result Corrected graph convolutions can improve node classification performance exponentially.
Gradient Gating improves deep GNNs by modulating message passing updates.
problem Oversmoothing and performance degradation in deep GNNs.
method Gradient gating mechanism for multi-rate message passing.
result G2 framework alleviates oversmoothing and achieves state-of-the-art performance. Graph Neural Networks (GNNs) have been emerging as a promising method for relational representation including recommender systems. However, various challenging issues of social graphs hinder the practical usage of GNNs for social recommendation, such as their complex noisy connections and high heterogeneity. The oversm…
A new method recovers latent potentials from graph flows, preserving ordering and stability.
problem Recovering latent potentials from graph flows is ill-posed and standard methods collapse the ordering.
method Gauge-invariant, parameter-insensitive regularization using Dirichlet energy.
result The method preserves ordering and stability across different regularization strengths.
Ripple Walk Training tackles graph neural network training issues for large and deep graphs.
problem Neighbors explosion, node dependence, and oversmoothing in large and deep GNNs.
method Subgraph-based training framework with Ripple Walk Sampler for high-quality subgraph sampling.
result RWT improves training efficiency and reduces space complexity for deep and large GNNs.
Graph neural networks are explained through heat diffusion analogy.
problem Limitations of graph neural networks (oversmoothing, oversquashing).
method Analogizing message passing in GNNs to heat dynamics.
result Fundamental understanding of GNNs and improved model design.
Medical image reconstruction is typically an ill-posed inverse problem. In order to address such ill-posed problems, the prior distribution of the sought after object property is usually incorporated by means of some sparsity-promoting regularization. Recently, prior distributions for images estimated using generative …
STAG injects noise into graph neural networks to improve performance.
problem Graph neural networks suffer from over-smoothing and limited discrimination.
method Introduces a stochastic aggregation framework (STAG) with adaptive noise injection.
result STAG models correct both over-smoothing and discrimination issues.
SCQRNN prevents quantile crossing and improves computational efficiency.
problem Quantile crossing issue in regression models.
method Integrates ad hoc sorting in training to prevent quantile crossing and enhance computational efficiency.
result SCQRNN achieves faster convergence and non-intersecting quantiles.
GRAND treats GNNs as PDE discretizations, addressing graph learning issues.
problem Graph learning issues like depth, oversmoothing, and bottlenecks.
method Models GNNs as a continuous diffusion process, treating them as PDE discretizations.
result Linear and nonlinear versions of GRAND achieve competitive results on graph benchmarks.
The paper analyzes Tikhonov regularization in Hilbert scales for statistical inverse problems.
problem Statistical inverse problems in Hilbert scales with general noise.
method Tikhonov regularization scheme with conditional stability estimates and high probability error bounds.
result Explicit rates of convergence for oversmoothing and regular cases over defined regularity classes.
GSAN learns adaptive node representations using geometric scattering and attention.
problem Oversmoothing in node representation learning.
method Attention-based architecture integrating geometric scattering and GCN channels.
result GSAN outperforms previous networks in semi-supervised node classification.
New mechanism for pure differential privacy on functional summaries using Laplace-like process.
problem Challenges in achieving differential privacy for complex, structured functional summaries.
method Independent Component Laplace Process (ICLP) mechanism for infinite-dimensional Hilbert space.
result Effective enhancement of utility of private summaries through oversmoothing.
Despite the wide application of Graph Convolutional Network (GCN), one major limitation is that it does not benefit from the increasing depth and suffers from the oversmoothing problem. In this work, we first characterize this phenomenon from the information-theoretic perspective and show that under certain conditions,…
A new Gaussian process framework uses neural feature maps for scalable, accurate inference.
problem Efficient and accurate Gaussian process inference for diverse data types.
method Neural feature maps to construct expressive kernels, with theoretical guarantees and practical scalability.
result The approach outperforms existing methods in accuracy and efficiency across various data modalities.
Many immunization strategies have been proposed to prevent infectious viruses from spreading through a network. In this study, we propose efficient immunization strategies to prevent a default contagion that might occur in a financial network. An essential difference from the previous studies on immunization strategy i…
New method combines regional HIV prevention trial data without sharing individual patient info.
problem Regional differences in HIV prevention efficacy, privacy concerns, and data sharing limitations.
method Federated learning approach that combines site-specific estimators via L1-regularization.
result Improved precision in estimating region-specific survival curves.
Modified model prevents volatility from approaching zero.
problem Volatility in the Gatheral model can approach zero, making it statistically indistinguishable.
method Proposed a modified model with Skorokhod reflection to prevent volatility from approaching zero.
result The modified model prevents volatility from approaching zero, preserving the model's flexibility.
This paper examines how skip connections prevent rank collapse in sequence models.
problem Rank collapse in sequence models, leading to reduced expressivity and training instabilities.
method Analytical and ablation studies of lambda-skip connections in SSMs.
result A sufficient condition to prevent rank collapse across various architectures.
Prevents sensitive data generation in diffusion models using labeled and unlabeled data.
problem Generating sensitive data in diffusion models using unlabeled data.
method Positive-Unlabeled Diffusion Models, approximating ELBO with labeled and unlabeled data.
result Prevents the generation of sensitive data without compromising image quality.
Deep Neural Networks are robust to minor perturbations of the learned network parameters and their minor modifications do not change the overall network response significantly. This allows space for model stealing, where a malevolent attacker can steal an already trained network, modify the weights and claim the new ne…
StratLearner learns strategies to prevent misinformation in social networks.
problem Learning strategies to protect against misinformation in social networks without knowing the diffusion model.
method Structured prediction framework using random features and large margin method.
result Our method produces near-optimal protectors without diffusion model information and outperforms other methods.
RAMBO optimizes multi-regime problems by discovering and modeling distinct energy basins.
problem Multi-regime problems in molecular conformation and drug discovery.
method Dirichlet Process Mixture of Gaussian Processes with adaptive hyperparameters and concentration parameters.
result Consistent improvements over state-of-the-art on multi-regime objectives.
Mobile apps and machine learning improve malaria prevention and treatment.
problem High malaria cases and deaths in low-income countries.
method Adaptive interventions using mobile health apps and machine learning.
result Increased malaria testing, adherence, and provider skills.
MFNs parameterize non-local interactions through matrix equivariant functions, improving graph neural network performance.
problem Challenges in modeling non-local interactions in graphs, such as oversmoothing and oversquashing.
method Matrix Function Neural Networks (MFNs) using resolvent expansions for non-local interactions.
result Achieves state-of-the-art performance in graph benchmarks and captures intricate non-local interactions.
New condition prevents hyperbolic spaces from matching curve complexes.
problem Identifying when hyperbolic spaces cannot match curve complexes.
method Analyzing specific hyperbolic complexes and identifying a condition.
result Identified a condition preventing quasi-isometry between hyperbolic spaces and curve complexes.
TAMD prevents degeneracy in finite mixtures, offering strong guarantees but modest practical improvements.
problem Degeneracy in maximum likelihood estimation of finite mixtures.
method Transcendental regularization with analytic barrier functions.
result Strong theoretical guarantees (identifiability, consistency, robustness) but modest practical improvements.
Large learning rates prevent memorization in denoising score matching.
problem Memorization of training data in diffusion-based generative models.
method Investigating the role of large learning rates in the small-noise regime, proving that they prevent convergence to the empirical optimal score.
result Large learning rates prevent memorization by making it impossible for the learned score to be arbitrarily close to the empirical optimal score.