Graph machine learning lacks a balanced theory, focusing on expressive power and optimization.
problem Insufficient theoretical understanding of GNNs' generalization behavior.
method Develop a balanced theory focusing on expressive power, generalization, and optimization.
result Theoretical advancements need to align with practical success in graph machine learning.
Theoretical analysis improves understanding of Deep Q-Learning's behavior.
problem Lack of formal guarantees and gaps between theory and practice of Deep Q-Learning.
method Dynamical systems perspective, focusing on realistic assumptions.
result Proves convergence of Deep Q-Learning under specific conditions.
Framework for understanding overfitting and underfitting using information theory.
problem Understanding and preventing overfitting and underfitting in machine learning.
method Information-theoretic framework measuring algorithm capacity and dataset information transfer.
result Upper-bounding algorithm capacity and establishing its relationship to machine learning quantities.
Encoder-decoder networks using convolutional neural network (CNN) architecture have been extensively used in deep learning literatures thanks to its excellent performance for various inverse problems. However, it is still difficult to obtain coherent geometric view why such an architecture gives the desired performance…
The study improves theoretical understanding of using multiple synthetic datasets for better model accuracy.
problem Lack of theoretical understanding of using multiple synthetic datasets for supervised learning.
method Derive bias-variance decompositions for multiple synthetic datasets settings.
result A simple rule of thumb to select the appropriate number of synthetic datasets.
In this paper we develop some group theoretical methods which are shown to be very useful for a better understanding of the properties of the Riccati equation and we discuss some of its integrability conditions from a group theoretical perspective. The nonlinear superposition principle also arises in a simple way.
Theoretical framework for neural network compression using sparsity norms.
problem Understanding and quantifying compressibility and accuracy trade-offs in neural networks.
method Using sparsity-sensitive ℓ_q-norm to characterize compressibility and developing adaptive pruning algorithms.
result Theoretical relationship between network sparsity and compressibility with controlled accuracy degradation.
Unsupervised pre-training improves model generalization, but lacks theoretical understanding.
problem Lack of theoretical understanding of unsupervised pre-training's impact on model generalization.
method Introduces a novel theoretical framework to analyze and enhance generalization.
result Enhances understanding of unsupervised pre-training and fine-tuning, proposing a new regularization method.
The study reveals how adversarial perturbations can include class features for generalization.
problem Understanding why adversarial examples deceive neural networks and transfer between networks.
method A one-hidden-layer network trained on mutually orthogonal samples.
result Adversarial perturbations, even of a few pixels, contain sufficient class features for generalization.
New theory explains how equivariant self-supervised learning improves feature extraction.
problem Contrastive learning sacrifices useful features due to invariance to data augmentations.
method Information-theoretic perspective to understand E-SSL's generalization ability.
result Equivariant self-supervised learning creates synergy between equivariant and classification tasks.
New framework explains normalizing flows' power and limitations.
problem Understanding the expressive power and limitations of normalizing flows.
method Theoretical framework for well-conditioned coupling-based normalizing flows and volume-preserving flows.
result RealNVP is distributionally universal, but volume-preserving flows are not.
Despite the empirical success of the actor-critic algorithm, its theoretical understanding lags behind. In a broader context, actor-critic can be viewed as an online alternating update algorithm for bilevel optimization, whose convergence is known to be fragile. To understand the instability of actor-critic, we focus o…
The paper explores stability and generalization of deep GCNs.
problem Understanding the stability and generalization of deep GCNs from a theoretical perspective.
method Theoretical analysis of stability and generalization properties of deep GCNs.
result The stability and generalization of deep GCNs are influenced by the maximum absolute eigenvalue of the graph filter operators and the depth of the network.
This paper explains GNNs using graph signal denoising.
problem Understanding how GNNs work for node representation learning.
method Spectral graph convolutional networks and graph attention networks are analyzed from the perspective of graph signal denoising.
result GNNs implicitly solve graph signal denoising problems.
New theory explains contrastive learning via overlapping augmented views.
problem Lack of theoretical understanding of contrastive learning.
method Augmentation overlap perspective to improve downstream performance.
result Asymptotically closed bounds for downstream performance under weaker assumptions.
Study explores learning behavior of GFlowNets, revealing key mechanisms.
problem Lack of theoretical understanding of GFlowNets' learning dynamics.
method Rigorous theoretical investigation of four dimensions: convergence, sample complexity, implicit regularization, and robustness.
result Elucidates mechanisms underlying GFlowNet's learning dynamics, providing insights into performance factors.
Improves understanding of neural network predictions using influence functions.
problem Challenges in understanding neural network predictions.
method Utilized NTK theory to calculate influence functions for over-parameterized neural networks.
result Proved that the approximation error of IF can be arbitrarily small in the over-parameterized regime.
The paper provides convergence guarantees for VAEs using SGD and Adam.
problem Understanding theoretical convergence guarantees for VAEs.
method Derives non-asymptotic convergence rates for VAEs trained with SGD and Adam.
result Convergence rate of \(\mathcal{O}(\log n / \sqrt{n})\) with explicit hyperparameter dependencies.
The paper improves dropout's utility by reducing interactions in deep neural networks.
problem Over-fitting problem in deep learning.
method Game-theoretic interactions analysis and interaction loss.
result Interaction loss improves dropout's utility and boosts DNN performance.
Develops a new robustness criterion for VAEs and provides theoretical guarantees.
problem Lack of formalization for robustness in VAEs.
method Introduces r-robustness criterion and derives reconstruction margins. result Derives theoretical guarantees for VAE robustness.
Background: It is still an open research area to theoretically understand why Deep Neural Networks (DNNs)---equipped with many more parameters than training data and trained by (stochastic) gradient-based methods---often achieve remarkably low generalization error. Contribution: We study DNN training by Fourier analysi…
Theoretical work shows integrating coherent reasoning improves LLM performance and error correction.
problem Improving reasoning and error correction in large language models (LLMs) with few-shot prompting.
method Theoretical analysis and sensitivity experiments on transformer behavior with coherent reasoning and corrupted demonstrations.
result The transformer gains better error correction ability and more accurate predictions when coherent reasoning is integrated.
Develops a logifold structure for understanding datasets.
problem Understanding and classifying complex datasets.
method Local-to-global approach using measure-theoretical models.
result Improves accuracy in data classification problems.
By simulating the easy-to-hard learning manners of humans/animals, the learning regimes called curriculum learning~(CL) and self-paced learning~(SPL) have been recently investigated and invoked broad interests. However, the intrinsic mechanism for analyzing why such learning regimes can work has not been comprehensivel…
Paper analyzes how contrastive learning structures learned representations.
problem Understanding the structure of learned representations in contrastive learning.
method Kernel-based contrastive learning framework (KCL) and statistical dependency viewpoint.
result Theoretical upper bound and generalization error bound for KCL.
This paper explains how transformers learn from unstructured data in ICL.
problem Understanding how transformers learn from unstructured data in in-context learning.
method A simple transformer model with one or two attention layers and positional encoding is used to study the role of each component in ICL.
result A transformer with two attention layers and a look-ahead attention mask can learn from unstructured data.
Several recent works have empirically observed that Convolutional Neural Nets (CNNs) are (approximately) invertible. To understand this approximate invertibility phenomenon and how to leverage it more effectively, we focus on a theoretical explanation and develop a mathematical model of sparse signal recovery that is c…
Paper analyzes self-supervised image denoising with denatured data.
problem Understanding the performance of self-supervised image denoising with denatured data.
method Theoretical analysis and numerical experiments on a denoising algorithm.
result Theoretical analysis shows the algorithm finds desired solutions to the optimization problem.
The paper explores dual learning, a technique that improves machine translation and image transformation.
problem Understanding and improving dual learning's effectiveness and conditions.
method Theoretical analysis and algorithmic extension of dual learning.
result Multi-step dual learning boosts performance under mild conditions.
Recent studies show overparameterized neural networks behave like convex systems.
problem Understanding the behavior of overparameterized neural networks.
method Analysis of two-layer neural networks, focusing on restricted settings and neural tangent kernel space.
result Overparameterized neural networks behave like convex systems under certain conditions.
This work explores the generalization properties of diffusion models, providing theoretical and empirical insights.
problem Theoretical understanding of diffusion models' generalization capabilities remains underdeveloped.
method Theoretical exploration and quantitative analysis of generalization gaps in diffusion models.
result Established polynomially small generalization error (O(n−2/5+m−4/5)) for diffusion models, avoiding the curse of dimensionality. Paper proves convergence rates for Gaussian kernel ridge regression.
problem Understanding convergence rates for Gaussian kernel ridge regression.
method Establishes polynomial convergence rates for KRR with fixed hyperparameters.
result First polynomial convergence rates for Gaussian kernel ridge regression.
Study improves theoretical understanding of Bayesian deep learning for classification tasks.
problem Theoretical gap in understanding Bayesian approaches in deep learning for classification.
method PAC-Bayes bounds techniques and Spike-and-Slab priors for sparse deep learning.
result Established non-asymptotic results for prediction error, achieving minimax optimal rates.
Theoretical analysis shows MDMs can be efficient but not for all metrics.
problem Understanding the efficiency-accuracy trade-off of diffusion language models.
method Theoretical analysis of Masked Diffusion Model (MDM) using perplexity and sequence error rate as metrics.
result MDM achieves near-optimal perplexity but requires linear scaling for sequence error rate, highlighting efficiency-accuracy trade-offs.
Study improves understanding of non-differentiable penalties in high-dimensional settings.
problem Theoretical understanding of non-differentiable penalties like generalized LASSO and nuclear norm in high-dimensional settings.
method Proportional high-dimensional regime analysis with finite sample upper bounds on expected squared error.
result LO provides accurate estimation of out-of-sample risk in high-dimensional settings.
Despite their great success in practical applications, there is still a lack of theoretical and systematic methods to analyze deep neural networks. In this paper, we illustrate an advanced information theoretic methodology to understand the dynamics of learning and the design of autoencoders, a special type of deep lea…
The paper studies how to improve language model inference using particle filtering.
problem Understanding the accuracy-cost tradeoffs of inference-time methods for large language models.
method Introduces particle filtering algorithms like Sequential Monte Carlo (SMC) to study language model inference.
result Identifies criteria enabling non-asymptotic guarantees for SMC and fundamental limits faced by all particle filtering methods.
In this work, we investigate the use of three information-theoretic quantities -- entropy, mutual information with the class variable, and a class selectivity measure based on Kullback-Leibler divergence -- to understand and study the behavior of already trained fully-connected feed-forward neural networks. We analyze …
The VAE's reconstruction ability is studied using PAC-Bayes theory.
problem Understanding the performance of VAEs for unseen data.
method PAC-Bayes theory is applied to analyze VAE's reconstruction error.
result Generalization bounds on VAE's reconstruction error are provided.
Develops a simple model to understand learning curves for arbitrary power laws.
problem Lack of theoretical understanding of scaling laws in machine learning.
method Analyzes a toy model to determine if learning curves are universal or depend on data distribution.
result Determines that learning curves can exhibit n−β for arbitrary power β>0. Batch Normalization (BN) improves both convergence and generalization in training neural networks. This work understands these phenomena theoretically. We analyze BN by using a basic block of neural networks, consisting of a kernel layer, a BN layer, and a nonlinear activation function. This basic network helps us unde…
The paper analyzes frameworks for integrating sustainability into investment decisions.
problem Understanding how ESG factors influence investment choices.
method Examined and analyzed various theoretical frameworks including Behavioral Finance, Modern Portfolio, and Risk Management.
result Investors increasingly integrate ESG factors to optimize financial outcomes and societal goals.
Despite widespread interest and practical use, the theoretical properties of random forests are still not well understood. In this paper we contribute to this understanding in two ways. We present a new theoretically tractable variant of random regression forests and prove that our algorithm is consistent. We also prov…
This paper analyzes kNN convergence over feature transformations.
problem The curse of dimensionality affects kNN performance in transformed feature spaces.
method Developed a novel analysis on kNN convergence rates over transformed features, linking properties of the transformed space to raw feature space.
result Theoretical analysis explains why some feature transformations are better for kNN.
Multitask learning aims at solving a set of related tasks simultaneously, by exploiting the shared knowledge for improving the performance on individual tasks. Hence, an important aspect of multitask learning is to understand the similarities within a set of tasks. Previous works have incorporated this similarity infor…
This paper introduces a new bound to explain generalization in over-parameterized models.
problem Understanding why some over-parameterized models generalize well while others do not.
method PAC-Chernoff bounds and smoothness measures based on large deviation theory.
result Interpolators with smoother structures generalize better, according to the new theoretical framework.
This paper clarifies VAE's property through geometric and information-theoretic interpretations.
problem The transparency of VAE model is an underlying issue.
method Quantitative understanding of VAE through differential geometry and information theory.
result VAE can be mapped to an implicit isometric embedding with a scale factor derived from the posterior parameter.
Despite a lack of theoretical understanding, deep neural networks have achieved unparalleled performance in a wide range of applications. On the other hand, shallow representation learning with component analysis is associated with rich intuition and theory, but smaller capacity often limits its usefulness. To bridge t…