arXiv research

Quadratic models explain neural network behavior during training.

Complex-valued neural networks avoid spurious local minima.

Recently, deep learning has achieved huge successes in many important applications. In our previous studies, we proposed quadratic/second-order neurons and deep quadratic neural networks. In a quadratic neuron, the inner product of a vector of data and the corresponding weights in a conventional neuron is replaced with…

Sharp asymptotics reveal how network width controls learnability in quadratic neural networks.

Paper presents an ADMM-based approach to efficiently integrate quadratic programming layers into neural networks.

Study on neural network dynamics in high dimensions with quadratic activation.

Despite their practical success, a theoretical understanding of the loss landscape of neural networks has proven challenging due to the high-dimensional, non-convex, and highly nonlinear structure of such models. In this paper, we characterize the training landscape of the mean squared error loss for neural networks wi…

New method improves neural network robustness by identifying functions rather than parameters.

Study on neural networks with quadratic activation functions, focusing on optimization and generalization.

Recent theoretical work has established connections between over-parametrized neural networks and linearized models governed by he Neural Tangent Kernels (NTKs). NTK theory leads to concrete convergence and generalization results, yet the empirical performance of neural networks are observed to exceed their linearized …

Bayes-optimal learning of a neural network with quadratic activations is achieved with GAMP-RIE.

Quantification of the stationary points and the associated basins of attraction of neural network loss surfaces is an important step towards a better understanding of neural network loss surfaces at large. This work proposes a novel method to visualise basins of attraction together with the associated stationary points…

Bayesian neural networks are shown to be minimax and admissible under certain conditions.

Study shows neural networks trained with GD converge to Gaussian processes with polynomial decay.

We show that the gradient descent algorithm provides an implicit regularization effect in the learning of over-parameterized matrix factorization models and one-hidden-layer neural networks with quadratic activations. Concretely, we show that given $\tilde{O}(dr^{2})$ random linear measurements of a rank $r$ positive s…

We study the supervised learning problem under either of the following two models: (1) Feature vectors ${\boldsymbol x}_i$ are $d$-dimensional Gaussians and responses are $y_i = f_*({\boldsymbol x}_i)$ for $f_*$ an unknown quadratic function; (2) Feature vectors ${\boldsymbol x}_i$ are distributed as a mixture of two $…

Study how generalization scales with model size and data in quadratic neural networks.

ResUNet-CMB neural network reconstructs CMB effects from noisy data.

We analyze single-layer neural networks with the Xavier initialization in the asymptotic regime of large numbers of hidden units and large numbers of stochastic gradient descent training steps. The evolution of the neural network during training can be viewed as a stochastic system and, using techniques from stochastic…

Extends quadratic loss for SVM and deep learning to improve pattern correlation.

New method reduces training time for deep hedging networks.

Study on SGD dynamics and scaling laws for training quadratic neural networks in high dimensions.

New SDP method certifies neural network robustness across all classes efficiently.

Improved stability analysis of neural network systems using Zames-Falb multipliers.

WildCat efficiently compresses neural network attention mechanisms.

We provide new theoretical insights on why over-parametrization is effective in learning neural networks. For a $k$ hidden node shallow network with quadratic activation and $n$ training data points, we show as long as $k \ge \sqrt{2n}$, over-parametrization enables local search algorithms to find a \emph{globally} op…

Deep neural networks enforce non-crossing quantile regression curves.

Study shows overparameterization helps shallow neural networks recover signals in high dimensions.

Bounds neural network output distribution to Gaussian for random initialization.

Efficiently solves heterogeneous QPs by reducing variables using instance-specific projections.

New MIP formulations for neural network Lipschitz constant estimation.

Neural networks can learn from higher-order cumulants efficiently, requiring quadratic samples.

Improved continual learning for neural networks with BN layers using K-FAC extension.

Simplifies neural network constraints with computationally efficient method.

Deep neural nets approximate high-dimensional HJB equations efficiently.

Increasing the batch size is a popular way to speed up neural network training, but beyond some critical batch size, larger batch sizes yield diminishing returns. In this work, we study how the critical batch size changes based on properties of the optimization algorithm, including acceleration and preconditioning, thr…

Improved uncertainty estimation in neural networks with VBLL.

QLA improves Bayesian uncertainty estimation for DNNs without increasing computational cost.

Catapult phase in neural nets shows exponential loss growth before quick decrease.

Neural network discovers exact solutions to QP with linear constraints.

The fundamental learning theory behind neural networks remains largely open. What classes of functions can neural networks actually learn? Why doesn't the trained network overfit when it is overparameterized? In this work, we prove that overparameterized neural networks can learn some notable concept classes, including…

Integrates prediction models into portfolio optimization for better asset allocation.

SGD efficiently learns the XOR function with near-optimal sample complexity.

Three-layer neural networks learn hierarchical polynomial functions efficiently.

The paper develops a convex parameterization for robust RNNs ensuring stability and robustness.

RFRBoost uses random features to boost deep residual neural networks, improving performance and computational efficiency.

We propose a novel end-to-end non-minimax algorithm for training optimal transport mappings for the quadratic cost (Wasserstein-2 distance). The algorithm uses input convex neural networks and a cycle-consistency regularization to approximate Wasserstein-2 distance. In contrast to popular entropic and quadratic regular…

HyCNNs improve convex function learning and optimal transport.