Survey on GNNs' power and limitations.
problem Theoretical limitations of GNNs.
method Comprehensive overview of GNNs and their variants.
result Provably powerful variants of GNNs.
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,657 papers · 148 categories
Trend · papers per month
48 results for “expressive power”
This paper explores how enforcing equivariance constraints limits neural network expressivity and proposes compensatory model size increases.
problem The impact of enforcing equivariance constraints on the expressive power of neural networks.
method Examined 2-layer ReLU networks, analyzed boundary hyperplanes and channel vectors, and constructed upper bounds on model size required for compensation.
result Enforcing equivariance constraints reduces the expressive power of neural networks, but this can be compensated by increasing model size.
kth-order invariant graph networks are as powerful as kth-order WL in distinguishing graphs.
problem Measuring the expressive power of graph neural network formalisms.
method Considered kth-order invariant graph networks (k-IGNs) and compared their expressive power to kth-order WL.
result k-IGNs and k-WL are equally powerful in distinguishing graphs.
The abstract discusses parallels between Galois theory and Stone-Weierstrass theorem in various fields.
problem Connecting distinguishing power and expressive power in different fields.
method Elementary theorem connecting distinguishing power and expressive power.
result Foundational principle in linguistics linking distinguishing power and expressive power.
New MPNNs match 2-WL, faster distinguishing graphs.
problem Improving graph neural network expressiveness.
method Introducing -walk MPNNs and second-order GNNs.
result Walk MPNNs match 2-WL and can distinguish graphs faster.
Study shows limits of certain normalizing flows in higher dimensions.
problem Understanding the representation power of normalizing flows in different dimensions.
method Rigorously established bounds on expressive power of basic normalizing flows.
result Limited representation power in higher dimensions, especially with moderate depth.
Study reveals Transformer's expressive power and mechanisms.
problem Understanding the approximation properties of Transformer for sequence modeling.
method Systematic study of Transformer's components and their combined effects, establishing approximation rates.
result Reveals roles of critical parameters in Transformer, such as number of layers and attention heads.
Geometric GNNs improve graph discrimination through GWL.
problem Discriminating geometric graphs embedded in Euclidean space.
method Proposed a geometric version of the Weisfeiler-Leman test (GWL) for geometric graphs.
result Characterized the expressive power of geometric GNNs based on physical symmetries.
Paper explores how GNNs can learn graph biconnectivity, finding ESAN is the only known expressive framework.
problem Understanding the expressive power of GNNs beyond the WL test.
method Introduces a novel class of expressivity metrics via graph biconnectivity and develops the GD-WL approach.
result GD-WL consistently outperforms prior GNN architectures in learning biconnectivity metrics.
GNNs with random node initialization are shown to be universally expressive.
problem Limitations of standard GNNs in distinguishing graphs.
method Random node initialization (RNI) to enhance GNNs' expressive power.
result GNNs with RNI are proven to be universally expressive.
PNCs balance tractability and expressiveness in probabilistic modeling.
problem Balancing tractability and expressiveness in probabilistic models.
method Introduce probabilistic neural circuits (PNCs) as a mix of Bayesian networks and neural networks.
result PNCs are powerful function approximators.
Deep neural networks can approximate complex functions through repeated compositions of a fixed-size ReLU network.
problem Understanding the expressive power of deep neural networks through function compositions.
method Demonstrated the surprising expressive power of repeated compositions of a single fixed-size ReLU network.
result Repeated compositions of a single fixed-size ReLU network can approximate 1-Lipschitz continuous functions on with an error .
Symbolic regression is a type of discrete optimization problem that involves searching expressions that fit given data points. In many cases, other mathematical constraints about the unknown expression not only provide more information beyond just values at some inputs, but also effectively constrain the search space. …
Graph machine learning lacks a balanced theory, focusing on expressive power and optimization.
problem Insufficient theoretical understanding of GNNs' generalization behavior.
method Develop a balanced theory focusing on expressive power, generalization, and optimization.
result Theoretical advancements need to align with practical success in graph machine learning.
Improved bounds on neural network expressivity.
problem Understanding neural network expressivity and approximation capabilities.
method Improved bounds on the maximal number of linear regions of ReLU-networks.
result New insights into the expressivity of neural networks.
We study the expressive power of kernel methods and the algorithmic feasibility of multiple kernel learning for a special rich class of kernels. Specifically, we define \emph{Euclidean kernels}, a diverse class that includes most, if not all, families of kernels studied in literature such as polynomial kernels and radi…
We survey results on neural network expressivity described in "On the Expressive Power of Deep Neural Networks". The paper motivates and develops three natural measures of expressiveness, which all display an exponential dependence on the depth of the network. In fact, all of these measures are related to a fourth quan…
Implicit models can match or exceed explicit models with more test-time compute.
problem Understanding the expressive power and scaling of implicit models.
method Nonparametric analysis of expressive power, mathematical characterization of implicit operators, and test-time scaling experiments.
result Implicit models can progressively express more complex mappings through iteration, matching a richer function class with test-time compute.
The paper calculates asymptotic expansions for specific types of oscillatory integrals.
problem Analyzing oscillatory integrals with complex phase functions.
method Using asymptotic expansions of simpler phase functions to derive results for more complex cases.
result Explicit computation of coefficients in asymptotic expansions for certain integrals.
We develop a general duality between neural networks and compositional kernels, striving towards a better understanding of deep learning. We show that initial representations generated by common random initializations are sufficiently rich to express all functions in the dual kernel space. Hence, though the training ob…
ESAN improves graph neural networks by processing subgraphs.
problem Limitations of MPNNs in graph isomorphism.
method ESAN represents graphs as sets of subgraphs and processes them with equivariant architectures.
result ESAN increases the expressive power of GNNs and more expressive architectures.
Enhances quantum machine learning models using Fock states.
problem Data-embedding bottleneck in quantum machine learning.
method Photonic-based bosonic data-encoding scheme in Fock space.
result Controlled expressive power via photon number.
Graph Neural Networks (GNNs) have achieved much success on graph-structured data. In light of this, there have been increasing interests in studying their expressive power. One line of work studies the capability of GNNs to approximate permutation-invariant functions on graphs, and another focuses on the their power as…
We study deep neural networks with polynomial activations, particularly their expressive power. For a fixed architecture and activation degree, a polynomial neural network defines an algebraic map from weights to polynomials. The image of this map is the functional space associated to the network, and it is an irreduci…
Paper compares expressive power of GNNs, proving approximation guarantees for practical architectures.
problem Understanding the expressive power of Graph Neural Networks (GNNs).
method Theoretical framework comparing invariant and equivariant GNNs, proving approximation guarantees for practical architectures.
result Folklore Graph Neural Networks (FGNN) are the most expressive architectures for a given tensor order.
Fine-tuning normalization layers can reconstruct smaller networks.
problem Understanding the expressive power of fine-tuning normalization layers.
method Random ReLU networks and sparsified networks were fine-tuned to reconstruct target networks.
result Fine-tuning normalization layers can reconstruct networks that are times smaller.
We investigate graph neural networks for multi-relational data.
problem Understanding and improving graph neural networks for multi-relational data.
method Aligning Relational GCN and Compositional GCN with the Weisfeiler-Leman test to understand their expressive power and introduce a new -RN architecture.
result The -RN architecture overcomes the expressiveness limitations of Relational GCN and Compositional GCN.
Paper studies the expressivity of Convolutional Neural Networks (CNNs).
problem Understanding the expressivity of CNNs and their superiority in deep learning.
method Mathematical analysis of linear regions in one-layer and multi-layer ReLU CNNs.
result Deeper CNNs and CNNs have more expressivity per parameter than fully-connected NNs.
A wide variety of deep learning techniques from style transfer to multitask learning rely on training affine transformations of features. Most prominent among these is the popular feature normalization technique BatchNorm, which normalizes activations and then subsequently applies a learned affine transform. In this pa…
We study power expansions of the characteristic function of a linear operator in a -dimensional superspace . We show that traces of exterior powers of satisfy universal recurrence relations of period . `Underlying' recurrence relations hold in the Grothendieck ring of representations of $\GL(V)$. The…
Quantized neural networks can represent all fixed-point functions under certain conditions.
problem Expressive power of quantized neural networks under fixed-point arithmetic.
method Analyzing necessary and sufficient conditions for quantized networks to represent all fixed-point functions.
result Various popular activation functions satisfy the sufficient condition for representing all fixed-point functions.
New framework explains normalizing flows' power and limitations.
problem Understanding the expressive power and limitations of normalizing flows.
method Theoretical framework for well-conditioned coupling-based normalizing flows and volume-preserving flows.
result RealNVP is distributionally universal, but volume-preserving flows are not.
Graph Neural Networks outperform the Weisfeiler-Lehman algorithm in representation power.
problem Limited representation power of Graph Neural Networks compared to the Weisfeiler-Lehman algorithm.
method Algebraic analysis using eigenvalue decomposition of graph operators.
result Graph Neural Networks produce more discriminative representations than the Weisfeiler-Lehman algorithm.
Improves hypergraph link prediction by breaking symmetry.
problem Limited expressivity of GWL-1 algorithm in hypergraph link prediction.
method Preprocessing algorithm to identify and replace symmetry-inducing subhypergraphs with covering hyperedges.
result Improves expressivity of GWL-1, leading to better link prediction.
Transformers model contextual relations using probabilistic measures, revealing their expressive power.
problem Lack of clear understanding of Transformer's ability to model contextual relations.
method Introduced a measure-theoretic framework connecting softmax attention and entropy-regularized optimal transport.
result Transformer architectures can approximate arbitrary contextual relations, and the choice of normalization affects how these relations are represented.
While graph kernels (GKs) are easy to train and enjoy provable theoretical guarantees, their practical performances are limited by their expressive power, as the kernel function often depends on hand-crafted combinatorial features of graphs. Compared to graph kernels, graph neural networks (GNNs) usually achieve better…
New framework extends graph neural networks by improving expressiveness and space efficiency.
problem Limitations of existing graph neural networks, particularly in expressive power and space complexity.
method Proposed -FWL+, a flexible and powerful framework that combines -FWL and -FWL+.
result Demonstrated N-FWL, a practical and theoretically sound instance of -FWL+, achieving record-breaking results on ZINC-Subset.
New analysis shows over-squashing limits GNNs' power.
problem Understanding the expressive power of GNNs, especially under over-squashing.
method Quantitative measure of pairwise interactions in MPNNs, proving capacity requirements.
result Over-squashing hinders GNNs' ability to learn complex node feature functions.
This work characterizes topological descriptors of graph products and their expressive power.
problem Capturing multiscale structural information in graph products using topological descriptors.
method Analysis of various filtrations on graph products, including Euler characteristic and persistent homology.
result Persistent homology of graph products contains more information than individual graphs.
LoRA enhances model adaptability without increasing parameters.
problem Theoretical understanding of LoRA's adaptability.
method Theoretical analysis of LoRA's expressive power.
result LoRA can adapt any model to a smaller target model with a specific rank.
MoEs can efficiently model complex tasks with low-dimensionality and sparsity.
problem Understanding the theoretical foundations of MoEs for complex tasks.
method Systematic study of MoEs with two structural priors: low-dimensionality and sparsity.
result MoEs can approximate functions on low-dimensional manifolds and exhibit exponential structured tasks.
Normalizing flows provide a general mechanism for defining expressive probability distributions, only requiring the specification of a (usually simple) base distribution and a series of bijective transformations. There has been much recent work on normalizing flows, ranging from improving their expressive power to expa…
CW Networks leverage cell complexes to enhance GNNs, achieving state-of-the-art results on molecular datasets.
problem Graph Neural Networks struggle with long-range interactions and lack principled ways to model higher-order structures.
method CW Networks use cell complexes to decouple computational and input graph structures, enabling flexible hierarchical message passing.
result CW Networks achieve state-of-the-art results on molecular datasets.
Recently, the Weisfeiler-Lehman (WL) graph isomorphism test was used to measure the expressive power of graph neural networks (GNN). It was shown that the popular message passing GNN cannot distinguish between graphs that are indistinguishable by the 1-WL test (Morris et al. 2018; Xu et al. 2019). Unfortunately, many s…
Study evaluates capacity and trainability of parametrized quantum circuits.
problem Finding the best type of circuits for hybrid quantum-classical algorithms.
method Geometric structure of parameter space, effective quantum dimension, and circuit expressiveness.
result Identifies a transition in quantum geometry leading to decay of quantum natural gradient for deep circuits.
The paper investigates how Global Self-attention improves GCNs.
problem Improving expressive power and addressing overfitting in GCNs.
method Applying Global Self-attention mechanism over graph features.
result GSA mechanism enhances expressive power and mitigates overfitting and over-smoothing in GCNs.
The effect of data encoding on the expressive power of variational quantum machine learning modelsquant-ph
Quantum models can approximate any function if data encoding allows for a rich enough frequency spectrum.
problem Theoretical properties of quantum machine learning models, particularly their expressive power.
method Investigated how data encoding affects the expressive power of parametrized quantum circuits.
result Quantum models can access increasingly rich frequency spectra by repeating data encoding gates, potentially making them universal function approximators.
Paper explains why robust generalization is hard in deep learning models.
problem Difficulty in achieving robust generalization despite good training accuracy.
method Theoretical analysis of expressive power for deep neural networks.
result Expressive power of neural networks affects robust generalization.