A new test detects differences between two distributions without flow.
problem Detecting differences between two distributions without flow.
method Zero-flow discrepancy (ZFD) and zero-flow two-sample test (ZF2ST).
result ZF2ST can detect strong differences in structured distributions.
HOPE uses Hilbert space to deconstruct deep network representations.
problem Deconstructing learned representations in deep networks is challenging.
method Introduces Hilbert Operator for Progressive Encoding (HOPE) to deconstruct network weights.
result HOPE provides an unbiased approach to network compression and fine-tuning.
Framework synthesizes programs for simulating complex models and estimating parameters.
problem Parameter estimation for complex models requires manual encoding of fixed model structures.
method Combines LLMs for program synthesis with neural simulation-based inference.
result Identifies plausible model families from open-ended prompts with high accuracy.
A neural network estimates sampling distributions for hard problems where classical methods fail.
problem Bootstrap failure in estimating sampling distributions for specific statistics.
method Neural network trained on simulated datasets using pinball loss.
result Neural network attains 95% nominal coverage and 97% improvement over classical methods on four bootstrap-failure problems.
A game-theoretic framework identifies influential hyperparameters for neural networks.
problem Understanding which hyperparameters are most important for neural network performance.
method Employing Shapley Effects for global sensitivity analysis and Pareto front sets for identifying effective configurations.
result Reveals which hyperparameters are most influential for different objectives in neural networks.
GNet uses Gaussian processes for scalable, flexible neural networks.
problem Large-scale predictive modeling with high computational and storage costs.
method GNet employs Gaussian processes with nonparametric activation functions and a fast algorithm for efficient training and predictions.
result GNet achieves competitive performance across various test problems, including nonlinear function prediction and real-world data regression.
This paper improves neural network approximation for analytic functions with adjustable depth and width.
problem Approximating analytic functions using neural networks with depth and width parameters.
method Characterizes approximation rates as a joint function of width (N) and depth (L) for ReLU networks.
result Establishes upper bounds for analytic function approximation rates of O(N^(-CL^τ)) with τ influenced by N and L.
Monte Carlo method trains deep neural networks without gradients.
problem Vanishing and exploding gradients in backpropagation.
method Randomly mutate parameters, keep if loss decreases.
result Gradient-free method trains deep networks effectively.
This paper identifies drift Lipschitz budget K as key to diffusion policy expressivity and statistical trade-offs.
problem Understanding and maximizing the expressivity of diffusion policies while managing statistical limitations.
method Identifying drift Lipschitz budget K as central, quantifying expressivity and statistical behavior, proving lower bounds, and providing practical implementation guidelines.
result Balancing expressivity and statistical complexity yields a finite-sample performance gap, with rates depending on sample size and drift type.
Two-layer neural networks must be robust, even with arbitrary weights.
problem Proving the robustness of two-layer neural networks with arbitrary weights.
method Developed a new function-space covering method to prove the robustness law, replacing parameter-space covering.
result Proved the conjectured law for two-layer networks with arbitrary real weights, biases, and affine skip connections.
LiST improves neural network robustness and calibration without manual tuning.
problem Developing robust and calibrated neural networks simultaneously.
method Lipschitz Scaling Training (LiST) that iteratively adjusts the global Lipschitz constant.
result LiST yields an out-of-the-box calibrated network with competitive accuracy and robustness.
Generative model for TPPs using signatures and distributional discrepancies.
problem Limitations of signature methods for TPPs and lack of global sequence-level loss in neural models.
method Introduce interarrival embedding to lift jump paths to continuous paths of bounded variation, enabling signature methods for discrete event sequences. Develop sigTPP, a signature-based generative model trained on path-level loss.
result sigTPP achieves the best average rank across multiple metrics and outperforms or is within a standard error of the strongest baseline in 64% of dataset-metric pairs.
Study Gaussian-process limits of neural networks using tensor programs.
problem Understanding the behavior of neural networks as they approach infinite width.
method Quantitative analysis through tensor programs and Wasserstein distance.
result Explicit finite-width error bounds, showing convergence to Gaussian-process limits.
Emputation learns imputation models guided by missingness assumptions.
problem Learning imputation models for missing data given observed data.
method Guided by specific missingness assumptions, Emputation trains a deep generative model to learn the extrapolation distribution of missing variables.
result The population minimizer of the emputation risk recovers the target extrapolation distribution under various identification assumptions.
ManifoldFlow relaxes fixed-spectrum Stiefel layers to learn a positive spectrum.
problem Fixed-spectrum Stiefel layers impose rigid spectral constraints.
method Introduces ManifoldFlow, a relaxation that learns a positive spectrum while keeping the basis on the Stiefel manifold.
result Learnable SPD spectrum improves performance in various settings.
Deep learning solves dynamic programming with recursive utility.
problem Challenges in solving high-dimensional discrete-time dynamic programming problems with recursive utility.
method Certainty Equivalent Learning (CEL) algorithm that learns certainty-equivalent value directly with neural networks.
result Accurate value and policy approximations in high-dimensional problems, comparable to VFI in some cases.
New SDP method certifies neural network robustness across all classes efficiently.
problem Certifying robustness of neural networks across multiple classes.
method Quadratic model + SDP relaxation + pruning strategy.
result Significant computational speed-up and scalability to large datasets.
Proposes a variational approach to shallow neural networks, bypassing optimization.
problem Theoretical understanding and optimization of shallow neural networks.
method Replaces discrete training with a continuum variational surrogate, proving global well-posedness and regularity.
result Optimal parameter density can be obtained by solving a single linear system, achieving O(1/N) generalization error. Proposes a new neural head for asymmetric representation learning.
problem Asymmetric representation learning in directed relations.
method Role-aware neural convex divergence head.
result Role-aware projections improve directional accuracy over plain ICNN-Bregman heads.
Neural networks estimate time-varying parameters in AR(p) models with different noise types.
problem Forecasting time-dependent parameters in AR(p) processes with varying noise.
method Deep learning for time-varying coefficients, Gaussian and Laplace noise models.
result Simple model with time-varying parameters can effectively forecast complex dynamics.
Neural network model predicts alternating event-free periods.
problem Dynamic prediction of alternating recurrent events with statistical nuance.
method Developed an online dynamic prediction framework using neural network theory.
result Outstanding performance in predicting alternating recurrent event-free time.
Study on the geometric Dyson Brownian motion of non-square matrix products.
problem Understanding the spectrum of a product of non-square random matrices.
method Proportional depth-width limit followed by mean-field limit, solving Burgers equation.
result Free log-normal law is obtained in the identity-start case.
Theoretical study explains grokking in neural networks.
problem Understanding the abrupt transition from fitting to generalizing in neural networks.
method Characterized a shell-core topological configuration of the solution space induced by Adam's optimization dynamics.
result Derived grokking scaling laws for learning rate, batch size, and regularization coefficient.
Neural networks can be simplified to linear regression for easier understanding by statisticians.
problem Introducing neural networks to statisticians who are not familiar with them.
method Describing neural networks that approximate linear regression and discussing customizations.
result Statisticians can now understand neural networks by focusing on linear regression.
A scalable Bayesian additive model for stellar flare detection using Gaussian process inference and hidden Markov models.
problem Bayesian time-series modeling for astronomical datasets
method Generative surrogate framework with Variational Autoencoder and neural network forward pass
result Significant reduction in computational time for stellar flare detection
INDEQS: A Graph-Based Neural Controlled Differential Equation Framework for Forecasting
problem Forecasting time series with neural networks
method Incorporating prior knowledge of a directed graph
result Outer informedness consistently improves forecasting accuracy
Superposition in autoencoders leads to loss in simple models.
problem Mechanistic interpretability of neural networks
method Analyzing mathematical basis for superposition and providing bounds for reconstruction loss
result Upper and lower bounds for L2 reconstruction loss in the sparse regime
Barren plateaus are not an average-case phenomenon, but a highly non-unique problem.
problem Avoiding barren plateaus in neural network training
method First-moment framework for initialization strategies
result Many families of inequivalent initialization strategies can avoid concentration
New activation functions achieve arbitrary-accuracy Sobolev approximation by fixed-size neural networks.
problem Approximation of Sobolev functions by neural networks
method Elementary Universal Activation Function and Differentiable Universal Activation Functions
result Arbitrary-accuracy Sobolev approximation by fixed-size neural networks
Neural networks can achieve optimal sample complexity for learning single-index models.
problem Achieving optimal computational-statistical tradeoff in learning Gaussian single-index models.
method Unified gradient-based algorithm for training a two-layer neural network, adaptable to various loss and activation functions.
result Sample complexity of ds⋆/2∨d matches the SQ lower bound up to a polylogarithmic factor. Unbiased gradient estimation for Markov chains
problem Estimating gradients of stationary means in Markov chains
method Propose new unbiased estimators
result Improves efficiency for slow mixing Markov chains
Data symmetries in neural networks can generate conserved quantities.
problem Conservation laws in neural networks
method Using tensorizable networks
result Data augmentation can induce conserved quantities
Generalization in deep neural networks can be analyzed using minimax rates for gradient methods.
problem Generalization performance of over-parameterized neural networks
method Establishing a connection between gradient-based methods and kernel methods
result Deriving minimax-optimal rates for GD and SGD under polynomial network width scaling
EML-CD discovers causal mechanisms from neural networks in a structured way.
problem Extracting causal mechanisms from neural network weights is ill-posed.
method Integrates EML operator into causal structure learning, representing each edge mechanism as a gated EML binary tree.
result Achieves SHD=11.2 +/- 0.4 on real data, matching or outperforming existing methods.
DiffSlack learns neural networks with nonlinear constraints via learnable slack variables.
problem Enforcing nonlinear inequality constraints in neural networks.
method DiffSlack reformulates inequalities as equalities with learnable slack variables, predicting them as part of the network output.
result DiffSlack achieves higher planning success rates and stronger geometric constraint satisfaction compared to existing methods.
Neural networks learn spectral representations for group composition.
problem Understanding structured emergence in neural network training.
method Lifting gradient flow to Fourier domain, proving convergence to irreducible representations.
result Neurons converge to single irreducible representations, cross-layer coefficients align.
Neural network model improves leaf spectral reflectance prediction for grapevines.
problem Inaccurate modeling of grapevine leaf spectral reflectance from traits.
method Multi-head attention neural network trained on grapevine-specific data.
result Model achieved high accuracy (R^2=0.84, NRMSE=1.52%) and outperformed PROSPECT-PRO.
Study efficient neural operator learning using variation spaces.
problem Operator learning using encoder-decoder neural networks.
method Introduce variation space for nonlinear operators, establish approximation bounds.
result Algebraic approximation and learning rates for polynomially decaying input and output encoding errors.
Neural Markov models improve time series analysis by balancing deep learning and classical models.
problem Modeling non-stationary time series with high data sparsity.
method Hybrid approach using neural networks to parameterize stochastic matrices, estimating time-inhomogeneous Markov chains.
result Reduction of Chapman-Kolmogorov discrepancy and superior likelihood in financial markets.
This study benchmarks likelihood-free inference methods for models with heavy-tailed or discrete data.
problem Comparing likelihood-free inference methods for models with structural features like heavy-tails or discreteness.
method Four approaches: MLE, NBE, EOT, and AW-NBE are evaluated using simulations.
result The choice of evaluation tools is crucial for models with extremes and discrete data.
Deep networks retain initial bias after training, affecting generalization.
problem Understanding how much initial bias in neural networks survives training.
method Introduced initialization memory to measure initial bias's survival.
result SGD can preserve initial bias, while Adam-family methods erase it.
Proposes a neural network method to combine nonprobability and probability survey samples.
problem Combining nonprobability and probability survey samples for accurate population mean estimation.
method Uses a deep neural network to estimate sampling scores from nonprobability samples and combines them with probability sample information.
result Proposed estimators improve robustness to parametric propensity-score misspecification, especially for nonlinear selection mechanisms.
Proposes DMOC for more nuanced neural network robustness.
problem Lipschitz continuity is too coarse for nuanced data-dependent behavior.
method Data-driven, architecture-agnostic framework based on DMOC.
result DMOC provides a finer notion of robustness relative to data distribution.
Optimizes insurance pricing by accounting for policyholders' price sensitivity.
problem Traditional insurance pricing does not consider policyholders' price sensitivity.
method Formulates insurance pricing as a decision-making problem and uses off-policy evaluation and stochastic control.
result Neural networks outperform existing techniques for policy optimization.
New saddle network architectures preserve convex-concave geometry in optimization problems.
problem Optimization models with convex x and concave y components.
method Structured separable decomposition and saddle network architectures.
result Proven one-dimensional approximation theorem and high accuracy on various test functions.
This work shows synthetic gradients can outperform backpropagation in sample efficiency.
problem The efficiency of backpropagation in training neural networks.
method Unified vectorized feedback framework for loss-based and reward-based learning, introducing synthetic gradients.
result Synthetic gradients can achieve lower gradient-estimation mean squared error than backpropagation under certain conditions.
Alignment of neural network representations is influenced by SNR and sample size.
problem Understanding how neural network representations align across different conditions.
method Controlled training of neural networks on perturbed datasets, analyzing alignment and generalization.
result Alignment varies monotonically with SNR but non-monotonically with sample size, with minimal alignment near the interpolation threshold.
Deep networks learn sparse hierarchical features without CoD.
problem Overparameterized deep networks struggle with the curse of dimensionality.
method Norm-constrained neural networks for sparse compositional functions.
result Deep networks can learn sparse hierarchical features efficiently.
The study analyzes how neural reward models learn features for policy optimization in a Gaussian single-index model.
problem Reward modeling in policy optimization and its impact on downstream value.
method Two-stage neural reward model: first learns hidden direction, then fits readout layer.
result For any feature-learning temperature above a dimension-free threshold, a constant fraction of neurons recover the hidden direction.
Model shows loss curve with two distinct exponents due to sparse activations.
problem Sparse activations impact neural network scaling laws.
method Introduced a model for neural scaling laws under sparse activations, derived asymptotic population loss, and analyzed gradient-descent dynamics.
result Loss curve exhibits double-descent peak near interpolation threshold with two distinct scaling exponents.
The paper provides non-asymptotic Edgeworth expansions for neural network outputs.
problem Approximating deviations of finite-width neural networks from their Gaussian limit.
method Multidimensional Edgeworth expansions of arbitrary order for neural network outputs.
result Established a bound on the total variation distance between neural network output and its Edgeworth approximation.
New optimizer SF-NorMuon matches tuned AdamW across various horizons.
problem Fixed learning-rate schedules in neural network training lead to strong path dependence and costly re-tuning.
method Schedule-Free Spectral Optimization (SF-NorMuon)
result SF-NorMuon outperforms tuned AdamW on 125M and 772M parameter models across different horizons.
Partial fusion combines neural networks to balance accuracy and efficiency.
problem Balancing accuracy and computational cost in neural networks.
method Extending weight aggregation methods based on neuron-level similarity, using partial optimal transport to match similar neurons.
result Achieves a flexible tradeoff between computational cost and performance.
Uniform bounds for neural network convergence without strong convexity assumptions.
problem Understanding the convergence of neural networks in the feature-learning regime.
method Establishing uniform-in-time weak propagation-of-chaos via mean-field deterministic Wasserstein-gradient-flow dynamics.
result Uniform bounds on the difference between infinite-width and finite-width neural network outputs, showing that fewer neurons can achieve a desired loss.
New metric explains neural network performance, simplifying generalization error calculation.
problem Precise characterization of neural network generalization error.
method Introducing Representation Gap, linking to intrinsic dimension and equivariant diffusion models.
result Asymptotic equivalent of Representation Gap is governed by intrinsic dimension, easy to estimate.
Dropout schedules can be optimized to significantly reduce model test loss.
problem Improving model performance in neural networks.
method Developed a mean-field theory of dropout at the edge of chaos, proposing front-loaded dropout schedules.
result Front-loaded dropout schedules reduce test loss by 18-35% over constant dropout.
New model predicts weekly earthquakes with better tail risk assessment.
problem Violation of Poisson assumption in seismic data.
method Neural network for per-cell overdispersion estimation.
result 8.6% reduction in mean pinball deviation, 12.5% lower CRPS in tail events.
New framework explains neural network behavior through geometric postulates.
problem Understanding neural network mechanisms and making them more transparent.
method Introducing the Pursuit of Subspaces (PoS) hypothesis as an axiomatic framework.
result Unified geometric perspective on neural network representation, computation, and generalization.
FLASH-MAX predicts electromagnetic fields from sparse data in seconds.
problem Predicting homogeneous electromagnetic fields from sparse pointwise observations.
method Exact-by-construction neural network architecture that satisfies Maxwell's equations symbolically.
result FLASH-MAX achieves sub-1% relative validation error from 1K sparse observations in seconds.
Symmetrizes loss functions to improve neural network robustness against noisy labels.
problem Designing robust loss functions for noisy labels in neural networks.
method Symmetrization of multi-class loss functions, focusing on cross-entropy and unhinged loss.
result The multi-class unhinged loss is the unique convex symmetric loss under suitable assumptions.