GroupSort neural networks can approximate Lipschitz continuous functions.
problem Understanding and improving the expressive power of neural networks with Lipschitz constraints.
method Introduced and studied GroupSort neural networks with constraints on weights, proving their ability to approximate Lipschitz continuous functions.
result GroupSort networks can represent any Lipschitz continuous piecewise linear functions and are well-suited for approximating general Lipschitz continuous functions.
Wasserstein active regression improves estimation precision.
problem Improving regression model accuracy through active learning.
method Combines Wasserstein distance and GroupSort Neural Networks for uncertainty quantification.
result Wasserstein active regression often provides more precise estimations.
Training neural networks under a strict Lipschitz constraint is useful for provable adversarial robustness, generalization bounds, interpretable gradients, and Wasserstein distance estimation. By the composition property of Lipschitz functions, it suffices to ensure that each individual affine transformation or nonline…
WGANs improve probability distribution approximation with depth and width trade-offs.
problem Approximating complex probability distributions accurately.
method Wasserstein GANs with GroupSort discriminators, quantified generalization bound.
result High-capacity discriminators are crucial for WGANs' performance.
A toolkit for path-norms enhances neural network generalization bounds.
problem Establishing generalization bounds for modern neural networks.
method Introducing a comprehensive toolkit for path-norms in ReLU networks with various operations.
result Established generalization bounds for modern neural networks that are the most widely applicable and recover/beat the sharpest known bounds.
New approach to certifiably robust neural networks using Boolean function perspective.
problem Lack of principled understanding and certified robustness for ℓ ∞ \ell_\infty ℓ ∞ perturbations. method New perspective on Boolean functions, deriving impossibility results, and developing a unified Lipschitz network.
result Unified Lipschitz network that bypasses expressive power limitations and achieves better certified robustness.
Graphs of neural networks are represented to preserve symmetry, improving performance across various tasks.
problem Lack of equivariance in neural network representations of other neural networks.
method Represent neural networks as computational graphs and use graph neural networks to preserve permutation symmetry.
result Single model encodes diverse neural architectures, outperforming state-of-the-art methods.
Neural networks can approximate functions uniformly across various measures.
problem Universal approximation of functions across different probability measures.
method Proving neural networks are dense in Orlicz spaces, extending classical theorems.
result Neural networks uniformly approximate functions for weakly compact families of measures.
Optimal rates for shallow ReLU networks in nonparametric regression.
problem Approximating smooth and non-smooth functions with shallow ReLU networks.
method Analysis of shallow ReLU k ^k k neural networks, using variation norms and deep learning theory. result Optimal approximation rates for shallow ReLU networks in nonparametric regression.
Novel framework explains generalization in deep neural networks.
problem Understanding and improving generalization in deep neural networks.
method Topological Quantum Neural Networks as the semi-classical limit of Deep Neural Networks.
result Demonstrates that the perceptron, viewed as the semi-classical limit, achieves similar results to standard neural networks without training.
Investigates how neural network graph structure impacts predictive performance.
problem Lack of understanding between neural network graph structure and predictive performance.
method Developed relational graph representation to analyze neural networks, identifying a 'sweet spot' for improved performance.
result Identified a 'sweet spot' in relational graph structure that significantly improves neural network predictive performance.
Equivariant neural networks use symmetry to interpret complex data.
problem Interpreting and understanding the behavior of equivariant neural networks.
method Decompose layers into simple representations and analyze nonlinear activation functions.
result Equivariant neural networks can be interpreted using a filtration generalizing Fourier series.
Two new criteria help understand the advantage of deep neural networks.
problem Understanding the advantage of deepening neural networks.
method Proposed two new criteria to evaluate the expressivity of functions computable by deep neural networks.
result Increasing layers is more effective than increasing units in improving the expressivity of deep neural networks.
Paper benchmarks quantum neural networks against classical ones for binary classification tasks.
problem Comparing quantum neural networks with classical ones for binary classification.
method Evaluated with two toy examples, focusing on model complexity and training data size.
result EQNN and QNN outperform ENN and DNN for smaller parameter sets and training data samples.
Secret neural networks hidden within trained models.
problem Excess capacity in neural networks allows embedding secret models.
method Novel framework for hiding secret neural networks within carrier networks.
result Detection of hidden networks is computationally infeasible.
The paper proves consistency of neural networks with regularization.
problem Overfitting in neural networks with large scale data.
method Theoretical framework of neural networks with regularization, sieves method, and minimal neural networks theory.
result The estimated neural network converges to the true underlying function as sample size increases.
New proof shows neural networks can represent all multivariate functions.
problem Representing all multivariate functions with neural networks.
method Proved that three-layer neural networks can represent both continuous and discontinuous functions.
result Three-layer neural networks can represent all multivariate functions, including discontinuous ones.
New learning rules for wide neural networks without backpropagation.
problem Training wide neural networks efficiently and without backpropagation.
method Input-weight alignment driven by gradient descent in the NTK regime.
result Biologically-motivated learning rules equivalent to backpropagation in wide networks.
DeepWeightFlow generates diverse neural network weights efficiently.
problem Generating complete neural network weights efficiently and accurately.
method Flow Matching in weight space with Git Re-Basin and TransFusion.
result DeepWeightFlow generates high-accuracy neural networks without fine-tuning.
Neural networks improve nonparametric regression with measurement errors.
problem Nonparametric regression with measurement errors.
method Proposes a neural network design using FNN, normalizing flow, and inference network.
result Neural network approach is more flexible and superior or comparable to classical methods.
Graph Metanetworks process diverse neural architectures efficiently.
problem Processing diverse neural architectures efficiently.
method Builds metanetworks using graph neural networks to process graphs representing input neural networks.
result Proves GMNs are expressive and equivariant to parameter permutation symmetries.
A powerful network teaches a weak one, improving its performance.
problem Improving the performance of a weak neural network using a more powerful one.
method During training, a weak network learns features from a strong network to minimize feature distance.
result A weak neural network can increase its performance by learning from a more powerful network.
Affine spiking neural networks learn efficiently and generalize well.
problem Learning with spiking neural networks, especially with positive weights.
method Affine encoders and decoders, continuous parameter dependence, gradient-based training.
result Affine spiking neural networks can approximate shallow ReLU networks and generalize well.
Unified platform SOCRATES for neural network analysis.
problem Analyzing neural networks for bugs and fairness.
method Standardized format, assertion language, and multiple analysis algorithms.
result Unified platform for neural network analysis.
Explains equivariant neural networks for machine learning.
problem Understanding equivariance in neural networks.
method Simple mathematical treatment of neural network concepts.
result Clarifies the mathematical basis of equivariant neural networks.
Research on neural networks has gained significant momentum over the past few years. Because training is a resource-intensive process and training data cannot always be made available to everyone, there has been a trend to reuse pre-trained neural networks. As such, neural networks themselves have become research data.…
Neural networks are mathematically represented via quiver representations.
problem Understanding how neural networks process data and create representations.
method Representing neural networks as quiver representations with activation functions.
result Neural networks' computations can be studied algebraically and geometrically.
Adapting robust statistics to neural networks, researchers found neural networks can be more robust with certain loss functions.
problem The robustness of neural networks in complex learning tasks.
method Adapting the regression breakdown point from robust statistics to neural networks and comparing different configurations and contamination settings.
result Neural networks can benefit from robust loss functions, as demonstrated in extensive simulations.
Paper introduces a noise-robust classification method using hypergraph neural networks.
problem Noisy label learning problem in image datasets.
method PCA for dimensionality reduction, then applies graph-based semi-supervised learning methods including hypergraph neural network.
result Our proposed hypergraph neural network achieves the best performance when noise level increases.
Paper introduces new neural network models and theories.
problem Understanding neural networks beyond over-parameterized regime.
method Develops two exact models and a novel representor theory.
result Provides insights into neural network training and kernel evolution.
New neural networks combine additive regression with traditional architectures.
problem Performance limitations and high parameter requirements of traditional neural networks.
method Introduce hybrid deep additive neural networks with simpler activation and basis functions.
result Hybrid neural networks achieve better performance with fewer parameters.
QRPNNs use quaternion-valued recurrent correlation neural networks to solve cross-talk issues.
problem Cross-talk problem in QRCNNs.
method Combining non-local projection learning with QRCNNs.
result QRPNNs exhibit greater storage capacity and noise tolerance.
New neural networks learn distribution functions using quantiles and moments.
problem Approximating functions of distributions in probability spaces.
method Quantile and moment neural networks, mixing quantile and moment features.
result Moment neural network outperforms others for bivariate distributions.
Quadratic models explain neural network behavior during training.
problem Understanding neural network dynamics during training with large learning rates.
method Developed and tested Neural Quadratic Models.
result Neural Quadratic Models exhibit the 'catapult phase' similar to neural networks.
Neural Ordinary Differential Equation (Neural ODE) has been proposed as a continuous approximation to the ResNet architecture. Some commonly used regularization mechanisms in discrete neural networks (e.g. dropout, Gaussian noise) are missing in current Neural ODE networks. In this paper, we propose a new continuous ne…
Probabilistic deep learning uses neural networks and models to handle uncertainty.
problem Handling uncertainty in deep learning models.
method Two approaches: probabilistic neural networks and deep probabilistic models.
result TensorFlow Probability library supports both approaches.
Braid theory optimizes neural network structures.
problem Optimizing the architecture of neural networks.
method Using braid theory to describe and construct neural network structures.
result Braid-based networks outperform other architectures in classification tasks.
Dense neural networks can't approximate all functions.
problem Approximation capabilities of dense neural networks.
method Model compression approach combining weak regularity lemma and graph neural networks.
result Existence of Lipschitz continuous functions not approximable by dense neural networks.
Quantum neural network and tensor network models outperform classical models in Japanese stock market predictions.
problem Improving stock return predictions using quantum and quantum-inspired machine learning.
method Evaluation of quantum neural network and tensor network models against classical models like linear and neural networks.
result Tensor network model outperforms classical models in Japanese stock market, including linear and neural network models.
Analysis of over-parameterized neural networks has drawn significant attention in recentyears. It was shown that such systems behave like convex systems under various restrictedsettings, such as for two-level neural networks, and when learning is only restricted locally inthe so-called neural tangent kernel space aroun…
Neural networks can separate non-separable data using feature maps.
problem Non-separable data in neural networks.
method Characterization of feedforward neural networks and use of feature maps.
result ReLU neural networks can separate concentric data.
Neural networks simplify SDR in regression tasks.
problem Sufficient dimension reduction in regression problems.
method Applying neural networks with rank regularization to estimate the central mean subspace.
result Neural networks effectively perform SDR, consistent with theoretical estimations.
Residual neural networks don't help overcome sampling complexity issues.
problem Learning invertible residual neural networks from samples is hard due to the curse of dimensionality.
method Investigated invertible residual neural networks and their sampling complexity.
result Invertible residual neural networks still suffer from the curse of dimensionality in sampling complexity.
Rational neural networks approximate functions more efficiently with less depth.
problem Choosing optimal nonlinear activation functions in neural networks.
method Rational activation functions with optimal bounds and efficiency proofs.
result Rational neural networks approximate smooth functions more efficiently than ReLU networks with exponentially smaller depth.
Recently low displacement rank (LDR) matrices, or so-called structured matrices, have been proposed to compress large-scale neural networks. Empirical results have shown that neural networks with weight matrices of LDR matrices, referred as LDR neural networks, can achieve significant reduction in space and computation…
New study shows neural networks need many samples for training.
problem How much data is needed to train a ReLU feed-forward neural network?
method Theoretical and empirical analysis of ReLU feed-forward neural networks.
result Generalization error scales at 1 / n 1/\sqrt{n} 1/ n in sample size n n n . Simplicial neural networks extend graph neural networks to handle higher-order interactions.
problem Handling higher-order interactions in complex data structures.
method Define a convolution operation for simplicial complexes and use it to construct convolutional neural networks.
result SNNs effectively impute missing data in coauthorship complexes.
Bayesian neural networks outperform calibrated neural networks for tabular data.
problem Uncertainty in neural network predictions for tabular data.
method Bayesian neural networks vs. post-hoc calibration methods.
result Bayesian neural networks yield competitive performance compared to calibrated neural networks.