Paper develops SINNOs for approximating stochastic processes.
problem Approximating stochastic processes with neural networks.
method Developed stochastic interpolation neural network operators (SINNOs) with random coefficients.
result Established boundedness, interpolation accuracy, and approximation capabilities of SINNOs.
This work combines recurrent models with diffusion for probabilistic time series forecasting.
problem Scalability and capturing high-dimensional distributions and cross-feature dependencies in time series forecasting.
method Combines recurrent neural networks' efficiency with diffusion models' probabilistic modeling, using stochastic interpolants and conditional generation.
result Offers scalable probabilistic time series forecasting methods.
Monotonic Linear Interpolation property in neural networks persists despite non-convexity.
problem Understanding the geometric properties of neural network loss landscapes.
method Tools from differential geometry to analyze the monotonicity of neural network weights.
result Sufficient conditions for the Monotonic Linear Interpolation property under mean squared error.
New method generates equilibrium glass configurations efficiently.
problem Sampling equilibrium configurations of amorphous materials is slow and difficult.
method Riemannian stochastic interpolation framework combining Riemannian stochastic interpolant and equivariant flow matching.
result Enforcing geometric and symmetry constraints significantly improves generative performance.
Stochastic gradient method converges as fast as deterministic for overparametrized models.
problem Convergence rate of stochastic gradient methods in overparametrized models.
method Proposes a regularity condition enabling fast convergence of SGD.
result Stochastic gradient method achieves the same convergence rate as deterministic gradient method.
The paper proposes a neural network method to calibrate LSV models without interpolation.
problem Calibrating LSV models with market option prices using neural networks.
method Parametrizing leverage function with neural networks and learning parameters from market prices; using deep hedging for variance reduction.
result The method accurately calibrates LSV models and outperforms interpolation methods.
In modern supervised learning, many deep neural networks are able to interpolate the data: the empirical loss can be driven to near zero on all samples simultaneously. In this work, we explicitly exploit this interpolation property for the design of a new optimization algorithm for deep learning, which we term Adaptive…
Randomly trained neural networks can generalize well if there's a simpler underlying teacher model.
problem Why randomly trained neural networks generalize well despite interpolating training data.
method Examined a random neural network that interpolates training data and showed it generalizes well if there's a simpler underlying teacher model.
result Randomly trained neural networks can generalize well if there's a simpler underlying teacher model.
VSDN models sporadic time series with neural SDEs.
problem Modeling irregular and sparse time series data.
method Variational Bayesian method and neural SDEs.
result VSDNs outperform state-of-the-art models in prediction and interpolation.
Bayesian interpolants explain neural network inferences concisely.
problem Understanding neural network inferences.
method Adapting Craig interpolants for neural networks.
result Produces precise, understandable explanations.
Deep neural networks can interpolate any dataset in the overparametrized regime.
problem Interpolating any dataset with deep neural networks in the overparametrized regime.
method Proving universal approximations and interpolating any dataset with deep neural networks, considering specific conditions on activation functions.
result Interpolation of any dataset is possible in the overparametrized regime with deep neural networks.
The study examines deep convolutional neural networks and their learning ability.
problem Understanding the learning ability of deep convolutional neural networks (DCNNs).
method Examines DCNNs under both underparameterized and overparameterized settings, using a novel network deepening scheme.
result Establishes the first learning rates of underparameterized DCNNs and shows how adding layers can create interpolating DCNNs with good learning rates.
Randomly sampled interpolators achieve zero generalization error with enough data.
problem Understanding the high generalization ability of machine learning models.
method Algebraic geometry tools to prove zero generalization error for random interpolators.
result Generalization error of randomly sampled interpolators becomes zero once the number of training samples exceeds a geometric threshold.
Survey of diffusion and optimal transport methods in machine learning.
problem Design and analysis of time-evolving probability distributions in machine learning.
method Switch from Eulerian to Lagrangian representation through vector fields.
result Both diffusion methods and optimal transport offer computational advantages.
Covariance shrinkage via stochastic interpolation
problem High-dimensional covariance estimation
method Recasting shrinkage as empirical risk minimization
result Reduces statistical risk through scheduling, flow maps, and early stopping
RNGI model bridges two probability densities on Riemannian manifolds efficiently.
problem Limited applicability of Euclidean stochastic interpolants to Riemannian manifolds.
method Introduces RNGI model interpolating between Riemannian manifold probability densities along geodesics.
result Proves temporal marginal density solves transport equation on Riemannian manifold.
New algorithm interpolates data with neural nets, independent of sample size.
problem Understanding neural networks' ability to memorize training data.
method Randomized algorithm for constructing interpolating neural networks.
result Guarantees that are independent of the number of samples, moving beyond worst-case memorization capacity bounds.
A semi-supervised framework using stochastic interpolation and latent representations.
problem Challenges in conditional generative modeling with scarce labeled data.
method Combines conditional stochastic interpolation with low-dimensional latent representations.
result Significantly improves sample complexity and achieves faster convergence rate.
The paper connects flatness to generalization in learning multi-index models with neural networks.
problem Understanding the generalization of non-convex neural networks using flatness measures.
method Analyzes 2-layer non-convex homogeneous neural networks and their connection to multi-index models.
result Flattest interpolators achieve small population loss and generalize well, establishing a direct link between flatness and generalization.
Proposes a neural network for calibrating stochastic volatility models.
problem Calibrating stochastic volatility models with robustness and efficiency.
method Combines grid approach with pointwise two-stage calibration, using random grids for training.
result Validates the approach with empirical and Monte Carlo experiments for rough Bergomi and Heston models.
This study uses neural networks to solve interpolation problems with sparse, infinitely wide layers.
problem Exact data interpolation using sparse, infinitely wide neural networks.
method Atomic norm framework to derive convex hulls and equivalent convex formulations.
result Simple characterizations of convex hulls for different constraints on network weights and biases.
Monotone neural networks can approximate and interpolate functions efficiently.
problem Understanding the efficiency and expressiveness of monotone neural networks.
method Solving the monotone interpolation problem using depth-4 networks and comparing size bounds with arbitrary networks.
result Monotone neural networks can approximate and interpolate functions efficiently, but may require exponential size in high dimensions.
The monotonic linear interpolation in deep networks often leads to plateaus, revealing biases in optimization.
problem Plateaus in the optimization landscape of deep networks during monotonic linear interpolation.
method Investigated monotonic linear interpolation on deep neural networks, focusing on biases in weights and biases.
result Interpolating weights and biases differently can lead to significant differences in loss and accuracy, revealing biases in optimization.
Enhances interpolation paths in latent space using particle filters.
problem Generating meaningful interpolations between data points in latent space.
method Introduces a discriminator network to guide particle filter sampling of interpolation paths.
result Improved variability and stronger drift towards high data density areas.
Over-parameterized neural networks generalize well in practice without any explicit regularization. Although it has not been proven yet, empirical evidence suggests that implicit regularization plays a crucial role in deep learning and prevents the network from overfitting. In this work, we introduce the gradient gap d…
Cubic spline smoothing improves interpolation between irregularly sampled data.
problem Interpolation discontinuity in recurrent neural networks for irregularly sampled sequences.
method Cubic spline smoothing compensation module trained end-to-end with ODE-RNN.
result Improves interpolation between irregularly sampled data points.
Deep ReLU networks need Ω(N) parameters to interpolate at irregularly spaced points.
problem Interpolating at irregularly spaced data points with deep ReLU networks.
method Analyzing the number of parameters required for interpolation.
result Ω(N) parameters are necessary for interpolation when δ is exponentially small in N.
Neural nets learn simple distributions first, then more complex ones.
problem Understanding how neural networks generalize from simple to complex functions.
method Stochastic gradient descent training, synthetic data, CIFAR10, ImageNet pre-training.
result Neural networks initially use lower-order statistics, then higher-order ones.
Proves a new law of robustness for interpolating arbitrary data distributions.
problem Understanding robust interpolation for arbitrary data distributions.
method Proves a Lipschitzness lower bound for robust interpolation.
result Demonstrates a two-fold law of robustness for interpolating functions.
The study tests inferences about neural network optimization from linear interpolation of loss landscapes.
problem Understanding the difficulty of neural network optimization problems.
method Linear interpolation of neural network loss landscapes, systematic evaluation of various factors.
result Linear interpolation does not correlate with model performance, challenging prior intuition.
Study reveals phase transition in neural networks near interpolation.
problem Understanding generalization and learning transitions in neural networks.
method Effective theory for approximating Bayes-optimal generalisation error.
result Unveils a discontinuous phase transition between universal and specialisation phases.
Neural networks generalize on simple data generated by a programming language.
problem Generalization of neural networks on low complexity data.
method Minimum description length (MDL) approach for feedforward neural networks.
result MDL feedforward neural networks generalize with high probability on simple data.
Analysis of SGD for Gaussian mixture classification using dynamical mean-field theory.
problem Learning dynamics of SGD for a neural network classifying Gaussian mixture.
method Applying dynamical mean-field theory to track SGD dynamics in high dimensions.
result Reveals how SGD navigates the non-convex loss landscape.
New SPS variant improves non-smooth optimization without small gradients.
problem Improving non-smooth optimization without small gradients.
method Safeguarded Stochastic Polyak Step Size (SPS s a f e _{safe} s a f e ) for non-smooth optimization. result Rigorous convergence guarantees for non-smooth convex optimization without strong assumptions.
Deep learning accelerates Monte Carlo SDE simulations with large time steps.
problem Accurate simulation of SDEs with large time steps.
method Polynomial chaos expansion with neural network learned stochastic collocation points.
result Data-driven scheme achieves strong convergence in Monte Carlo simulations.
Wide neural networks' last hidden layers split into groups of redundant neurons.
problem Understanding why wide neural networks generalize well despite overfitting.
method Analyzed the last hidden layer representations of various convolutional neural networks.
result Wide hidden layers split into groups of redundant neurons, which help generalize.
Unified ODE model explains residual and non-residual networks.
problem Unclear relationship between residual and non-residual networks.
method Introducing a damping term in an ODE model to interpolate between ResNet and CNN architectures.
result Unified framework for understanding residual and non-residual networks.
This paper finds sparsest ReLU networks for interpolating data.
problem Finding the sparsest neural network that fits a dataset.
method Proposes a continuous, differentiable objective function based on ℓ p \ell^p ℓ p quasinorms. result Global minimizers of the proposed objective correspond to sparsest ReLU networks.
Recent works have shown that stochastic gradient descent (SGD) achieves the fast convergence rates of full-batch gradient descent for over-parameterized models satisfying certain interpolation conditions. However, the step-size used in these works depends on unknown quantities and SGD's practical performance heavily re…
New study finds many neural networks are not benignly overfitting.
problem Understanding the behavior of overfitting in neural networks.
method Exploring kernel ridge regression and deep neural networks to identify overfitting behaviors.
result Many interpolating methods, including neural networks, exhibit tempered overfitting rather than benign or catastrophic.
We show that stochastic interpolation flow maps are Lipschitz with a sharp constant.
problem High dimensional sampling and transport problems.
method Investigating stochastic interpolation flow for generating data samples.
result Stochastic interpolation flow maps are Lipschitz with a sharp constant matching optimal transport maps.
Neural networks can interpolate random data but still generalize well, studied in the NT regime.
problem Understanding how neural networks interpolate random labels and generalize well in the overparametrized regime.
method Characterization of the eigenstructure of the empirical NT kernel and generalization error of NT ridge regression.
result The generalization error is well approximated by polynomial ridge regression with an increased regularization parameter.
In this paper we explore acceleration techniques for large scale nonconvex optimization problems with special focuses on deep neural networks. The extrapolation scheme is a classical approach for accelerating stochastic gradient descent for convex optimization, but it does not work well for nonconvex optimization typic…
Deep kernel learning combines the non-parametric flexibility of kernel methods with the inductive biases of deep learning architectures. We propose a novel deep kernel learning model and stochastic variational inference procedure which generalizes deep kernel learning approaches to enable classification, multi-task lea…
Neural networks learn incrementally from orthogonal data, interpolating with minimal complexity.
problem Understanding the learning dynamics and implicit bias in ReLU networks with orthogonal data.
method Gradient flow analysis of two-layer ReLU networks from small initialization with orthogonal training data.
result The learned interpolator has a squared ℓ 2 \ell_2 ℓ 2 -norm scaling as n \sqrt{n} n , close to the minimal interpolator's complexity. Strong inductive biases prevent harmless interpolation in overparameterized models.
problem Understanding the conditions under which overparameterized models can interpolate noise without overfitting.
method Theoretical analysis of high-dimensional kernel regression and deep neural networks, focusing on the role of inductive biases.
result The strength of an estimator's inductive bias determines whether interpolation is harmless or requires fitting noise for good generalization.
Two-layer neural networks can overfit without increasing risk when data is noisy.
problem Understanding why neural networks can overfit without increasing risk in noisy data.
method Combining bias and variance analysis in a high-dimensional setting.
result The excess learning risk of the interpolator decays under mild conditions.
Neural networks can interpolate noisy data and still generalize well.
problem Generalization of neural networks trained on noisy data.
method Two-layer neural networks trained to interpolation by gradient descent on corrupted labels.
result Neural networks can achieve zero training error and optimal test error.