Study on deep neural networks using concentration inequalities and optimal stopping.
problem Understanding the performance and structure of stochastic deep neural networks.
method Introduced concentration inequalities for SDNN outputs and an EC classifier. Determined the optimal number of layers via an optimal stopping procedure.
result Optimal number of layers for SDNNs determined via an optimal stopping procedure.
The paper solves a specific type of Ambrosetti-Prodi problem with solutions having clustering concentration layers.
problem Solving a particular Ambrosetti-Prodi type problem with solutions having concentration layers.
method Using a matrix function and eigenfunction of a related operator, the paper constructs solutions with concentration layers directed along a closed curve.
result Proves the existence of a sequence of solutions with clustering concentration layers directed along a closed curve.
Unified theory of ownership concentration, overlap, and dependence.
problem Understanding the complex layers of ownership concentration, overlap, and dependence in financial markets.
method Develops a unified quadratic framework for analyzing these layers and their interactions.
result Unified framework shows that the same residual operator measures static overlap and governs linearized market transmission.
Graph Neural Network (GNN) research has concentrated on improving convolutional layers, with little attention paid to developing graph pooling layers. Yet pooling layers can enable GNNs to reason over abstracted groups of nodes instead of single nodes. To close this gap, we propose a graph pooling layer relying on the …
Randomly initialized neural networks can linearly separate arbitrary sets.
problem Mapping two arbitrary sets to linearly separable sets.
method Randomly initialized one-layer neural networks with sufficient width.
result With high probability, these networks can transform two sets into linearly separable sets.
We study the concentration of NTK for MLPs at EOC, proving finite-width approximation of gradient independence.
problem Understanding the concentration of Neural Tangent Kernel (NTK) for MLPs at the Edge of Chaos (EOC).
method Proved approximate gradient independence holds at finite width, using maximal inequalities to show NTK matrix concentrates around its infinitely wide limit.
result The NTK matrix of MLPs at EOC concentrates around its infinitely wide limit, requiring hidden layer widths to grow quadratically.
Unified framework for measuring concentration in weighted networks considering both weight distributions and network structure.
problem Traditional indices neglect the topology of relationships among network elements.
method Develops a family of topology-aware concentration indices that jointly account for weight distributions and network structure.
result The proposed indices preserve key properties and allow concentration to be evaluated across different dimensions of dependence.
New method recovers signals from compressed measurements using generative networks with contractive layers.
problem Signal recovery from compressed measurements with generative network priors.
method Developed a new matrix concentration inequality (R2WDC) to relax expansivity conditions for generative networks.
result Signals in the range of a Gaussian generative network can be recovered from few linear measurements with contractive layers.
In this paper, we diagnose deep neural networks for 3D point cloud processing to explore utilities of different intermediate-layer network architectures. We propose a number of hypotheses on the effects of specific intermediate-layer network architectures on the representation capacity of DNNs. In order to prove the hy…
Gradient descent trains both layers of a ReLU network to fit a linear model.
problem Training dynamics of a ReLU network to fit a linear target function.
method Jointly training both layers of a one-hidden-layer ReLU network in a realizable setting with Gaussian inputs and labels.
result Gradient descent from a small random initialization converges to a global minimizer at a linear rate with optimal sample complexity.
Unified framework for studying softmax attention under large prompts.
problem Challenges in theoretical analysis of softmax attention.
method Measure-based framework for finite and infinite prompts.
result Softmax attention converges to linear attention in the large-prompt regime.
We provide a theoretical explanation of the role of the number of nodes at each layer in deep neural networks. We prove that the largest variation of a deep neural network with ReLU activation function arises when the layer with the fewest nodes changes its activation pattern. An important implication is that deep neur…
The paper studies neural networks with wide layers and finds a deformed semicircle law.
problem Investigating spectral distributions of neural networks in the ultra-wide regime.
method Analyzes empirical kernel matrices, proves deformed semicircle law, provides nonlinear Hanson-Wright inequality.
result Emergence of a deformed semicircle law in the ultra-wide neural network regime.
Proposes a new latent variable model for hyperspherical latent spaces.
problem Efficiently modeling heavy-tailed distributions in hyperspherical latent spaces.
method Introduces spherical Cauchy (spCauchy) latent variables and applies Möbius transformations.
result Shows spCauchy recovers vMF geometry in high-concentration limits and avoids complex evaluations.
When the parameters are independently and identically distributed (initialized) neural networks exhibit undesirable properties that emerge as the number of layers increases, e.g. a vanishing dependency on the input and a concentration on restrictive families of functions including constant functions. We consider parame…
We give a covering number bound for deep learning networks that is independent of the size of the network. The key for the simple analysis is that for linear classifiers, rotating the data doesn't affect the covering number. Thus, we can ignore the rotation part of each layer's linear transformation, and get the coveri…
Quantum neural networks converge to Gaussian processes as they grow.
problem Understanding the convergence of quantum neural networks to Gaussian processes.
method Analyzing Haar random unitary and orthogonal deep QNNs, considering input states, measurement observables, and non-independence of unitary matrix entries.
result Quantum neural networks outputs converge to Gaussian processes in the limit of large Hilbert space dimension.
This paper presents a phase diagram for two-layer neural networks under different initialization scales.
problem Understanding the behavior of neural networks under varying scales of initialization.
method Analysis of a phase diagram for two-layer neural networks.
result Condensation of weight vectors on isolated orientations during training.
PortBench benchmarks LLMs for PM, revealing their weaknesses in diversification and robustness.
problem Lack of benchmarks for LLM-driven portfolio management, especially in diversification and robustness.
method Developed a comprehensive benchmark with a static QA dataset and a dynamic allocation pipeline, introducing metrics to evaluate correlation and robustness.
result 90% of LLMs fail to outperform a basic equal-weight allocation, highlighting their limitations in diversification and robustness.
Piecewise linear activations create many spurious local minima in neural networks.
problem Understanding the loss surface of neural networks with piecewise linear activations.
method Proved the existence of infinite spurious local minima and partitioned the loss surface into smooth cells.
result Piecewise linear activations create many spurious local minima that are invariant under a continuous path.
Reduced-order model improves LES for atmospheric pollutant dispersion.
problem Accurate near-field pollutant concentration tracking in urban areas.
method Combining POD and GPR for non-intrusive reduced-order modeling.
result Component-by-component optimization captures spatial scales in high-order modes.
MTL improves multi-dimensional regression in luminescence sensing.
problem Challenges in modeling multi-dimensional regression problems with classical methods.
method Multi-task learning (MTL) with feed-forward neural networks (FFNNs).
result MTL allows predicting multiple parameters from a single set of measurements.
The paper explores how linear neural networks can overfit without bias when data is well-behaved.
problem Understanding why linear neural networks can generalize well despite fitting noisy data.
method Analyzing two-layer linear neural networks trained with gradient flow, deriving bounds on excess risk.
result The excess risk depends on initialization quality and data covariance matrix properties.
Softmax policy gradient achieves global optimality in wide neural networks with entropy regularization.
problem Optimizing softmax policies with neural networks in the mean-field regime.
method Modeling neural networks as Wasserstein gradient flows and proving global optimality of fixed points.
result Global optimality of softmax policy gradient in wide single hidden layer neural networks with entropy regularization.
Proposes deep graph persistence to address neural persistence issues in deep learning.
problem Variance of weights and lack of spatial structure in deep neural networks impact neural persistence.
method Extends neural persistence to the whole network, considering interactions between layers.
result Deep graph persistence alleviates variance-related issues and captures persistent paths through the network.
The paper analyzes tensor recovery from symmetric rank-one measurements using information theory.
problem Recovering tensors with low symmetric rank from symmetric rank-one measurements.
method Covering numbers argument, Carbery-Wright inequality, orthogonal polynomials, Fano's inequality.
result Near-optimal sample complexity bounds for log-concave distributions.
A new method reduces task interference in model merging.
problem Model merging overlooks structural information and is susceptible to task interference.
method Task Singular Vectors (TSV) and TSV-Compress for compression and interference reduction.
result TSV-Merge significantly outperforms existing methods.
We revisit the initialization of deep residual networks (ResNets) by introducing a novel analytical tool in free probability to the community of deep learning. This tool deals with non-Hermitian random matrices, rather than their conventional Hermitian counterparts in the literature. As a consequence, this new tool ena…
We develop a fast, tractable technique called Net-Trim for simplifying a trained neural network. The method is a convex post-processing module, which prunes (sparsifies) a trained network layer by layer, while preserving the internal responses. We present a comprehensive analysis of Net-Trim from both the algorithmic a…
A longstanding problem for Deep Neural Networks (DNNs) is understanding their puzzling ability to generalize well. We approach this problem through the unconventional angle of \textit{cognitive abstraction mechanisms}, drawing inspiration from recent neuroscience work, allowing us to define the Cognitive Neural Activat…
Analyzes neural networks using linear models to understand their behavior.
problem Understanding multi-layer neural networks through linear models.
method Recalls and reviews four models: linear regression with concentrated features, kernel ridge regression, random feature model, and neural tangent model.
result Highlights limitations of linear theory and discusses approaches to overcome them.
Bayesian neural networks improve uncertainty calibration without sacrificing accuracy.
problem Bayesian neural networks struggle with uncertainty calibration and high-dimensional geometry.
method Model uncertainty only in weight directions using a von Mises-Fisher posterior on the unit sphere, deriving a compact KL term.
result A lightweight, dimension-aware variational unit improves calibration without sacrificing accuracy.
Paper characterizes gradient descent dynamics for neural networks with finite width.
problem Characterize gradient descent dynamics for multi-layer neural networks.
method Non-asymptotic state evolution theory for finite-width networks.
result Gradient descent dynamics provide precise distributional characterization.
The paper tackles learning varying DAG structures based on contextual features.
problem Learning a single DAG for the entire population from observational data.
method A neural network that maps contextual features to a weighted adjacency matrix of a DAG, with a projection layer to ensure acyclicity.
result The new approach can recover context-specific DAGs where existing methods fail.
In natural language processing, a lot of the tasks are successfully solved with recurrent neural networks, but such models have a huge number of parameters. The majority of these parameters are often concentrated in the embedding layer, which size grows proportionally to the vocabulary length. We propose a Bayesian spa…
Theory for algebraic data on categories via concentration structures.
problem Defining algebraic structures on categories.
method Introducing concentration structures and concentration monoids.
result Every group can be represented as a concentration monoid of a trivial category.
Paper addresses concentration of distances for fractional quasi p-norms, identifying conditions for concentration and anti-concentration.
problem Understanding concentration of distances for fractional quasi p-norms in high dimensions.
method Analyzes conditions for concentration and anti-concentration of distances for fractional quasi p-norms.
result Identifies conditions for concentration and anti-concentration of fractional quasi p-norms, ruling out some approaches and specifying conditions for control.
Study Finsler metric measure manifolds' concentration properties.
problem Understanding concentration properties in Finsler metric measure manifolds.
method Established relationships with observable diameter, isoperimetric inequalities, and first eigenvalue.
result Derived a Cheng type upper bound estimate for the first closed eigenvalue.
Deep networks can perfectly classify two low-dimensional manifolds on a sphere with large depth and width.
problem Binary classification of two low-dimensional submanifolds on a sphere.
method Analysis of a deep fully-connected neural network trained to separate two submanifolds of the unit sphere.
result Randomly-initialized gradient descent can perfectly classify the two manifolds with high probability when the network depth is large relative to certain geometric and statistical properties of the data.
New method improves missing mass concentration bounds.
problem Missing mass concentration problem
method New method of estimating concentration of heterogenic sums
result Slightly improved state-of-the-art bounds
Surfaces in 3-manifolds concentrate at curvature critical points.
problem Understanding concentration of surfaces in 3-manifolds.
method Proving surfaces concentrate at critical points of scalar curvature.
result Simply connected H-surfaces concentrate at curvature critical points.
Sharp concentration bounds for i.i.d. variables.
problem Controlling the tail probabilities of independent variables.
method Extension of Sanov's theorem using large deviations and information theory.
result Matching concentration and anti-concentration bounds for i.i.d. samples of any size.
Multiplex Network Hawkes model for systemic risk measurement
problem Investigate how contagion in financial networks is affected by different transmission channels
method Multiplex Network Hawkes model
result Sparse contagion pathways, with systemic-risk transmission concentrated in outward flows from a small number of influential institutions
We survey recent results related to the concentration of eigenfunctions. We also prove some new results concerning ball-concentration, as well as showing that eigenfunctions saturating lower bounds for L1-norms must also, in a measure theoretical sense, have extreme concentration near a geodesic.
A new model CDTM improves text classification by concentrating document topics.
problem Unsupervised text classification with diverse topic distributions.
method Imposes an exponential entropy penalty on document topic distribution to encourage concentration.
result More coherent topics and concentrated, sparse document-topic distributions.
Simplified proof of Gaussian concentration inequality using covariance.
problem Gaussian concentration inequality proof
method Covariance representation based on characteristic functions
result Elementary proof of Gaussian concentration inequality
The paper explores how data geometry influences generalization in neural networks.
problem Understanding generalization in overparameterized neural networks.
method Theoretical exploration of overparametrized two-layer ReLU networks trained below the edge of stability.
result Generalization bounds adapt to the intrinsic dimension of data distributions and deteriorate as data concentrates towards the unit sphere.
Study provides bounds for estimating intrinsic dimension using Gaussian kernels.
problem Estimating intrinsic dimension from data.
method Finite-sample concentration and anti-concentration bounds for Gaussian kernel sums.
result Explicit dependence on sample size, bandwidth, and geometric parameters.