arXiv research

Paper connects MoE and self-attention, proposing active-attention.

Complex-valued neural networks avoid spurious local minima.

Bayes-optimal learning of a neural network with quadratic activations is achieved with GAMP-RIE.

Study on neural network dynamics in high dimensions with quadratic activation.

We show that the gradient descent algorithm provides an implicit regularization effect in the learning of over-parameterized matrix factorization models and one-hidden-layer neural networks with quadratic activations. Concretely, we show that given $\tilde{O}(dr^{2})$ random linear measurements of a rank $r$ positive s…

Study on neural networks with quadratic activation functions, focusing on optimization and generalization.

Sharp asymptotics reveal how network width controls learnability in quadratic neural networks.

Despite their practical success, a theoretical understanding of the loss landscape of neural networks has proven challenging due to the high-dimensional, non-convex, and highly nonlinear structure of such models. In this paper, we characterize the training landscape of the mean squared error loss for neural networks wi…

Full-batch GD outperforms one-pass SGD in learning a single-index model with quadratic activation.

Study active learning of PTFs with derivative access.

New algorithm tackles stochastic optimization with inequality constraints.

This paper extends the convergence rate of DEQs with ReLU to any general activation.

New analysis shows capacity of treelike neural networks with various activations.

We provide new theoretical insights on why over-parametrization is effective in learning neural networks. For a $k$ hidden node shallow network with quadratic activation and $n$ training data points, we show as long as $k \ge \sqrt{2n}$, over-parametrization enables local search algorithms to find a \emph{globally} op…

Polynomial neural networks explore thresholds for maximum expressiveness.

One-pass SGD dynamics in overparameterized quadratic networks show slow escape from poor solutions.

Efficiently selects nearest neighbors for labeling to speed up active learning.

Gradient descent dynamics in quadratic regression models are analyzed, revealing five phases: monotonic, catapult, periodic, chaotic, and divergent.

The problem of active diagnosis arises in several applications such as disease diagnosis, and fault diagnosis in computer networks, where the goal is to rapidly identify the binary states of a set of objects (e.g., faulty or working) by sequentially selecting, and observing, (noisy) responses to binary valued queries. …

We describe a framework for designing efficient active learning algorithms that are tolerant to random classification noise and are differentially-private. The framework is based on active learning algorithms that are statistical in the sense that they rely on estimates of expectations of functions of filtered random e…

Deep neural networks enforce non-crossing quantile regression curves.

Catapult phase in neural nets shows exponential loss growth before quick decrease.

Improved stability analysis of neural network systems using Zames-Falb multipliers.

Finding minimum distortion of adversarial examples and thus certifying robustness in neural network classifiers for given data points is known to be a challenging problem. Nevertheless, recently it has been shown to be possible to give a non-trivial certified lower bound of minimum adversarial distortion, and some rece…

We show LLMs can be locally linear, enabling better control of activations.

New SDP method certifies neural network robustness across all classes efficiently.

Over-parametrization speeds up learning a single neuron model.

New method speeds up neural kernel computations for various activations.

New lower bounds improve logistic log-likelihood optimization and inference.

Traditional human activity recognition (HAR) based on time series adopts sliding window analysis method. This method faces the multi-class window problem which mistakenly labels different classes of sampling points within a window as a class. In this paper, a HAR algorithm based on U-Net is proposed to perform activity…

New method improves neural network robustness by identifying functions rather than parameters.

Study on SGD dynamics and scaling laws for training quadratic neural networks in high dimensions.

We study the supervised learning problem under either of the following two models: (1) Feature vectors ${\boldsymbol x}_i$ are $d$-dimensional Gaussians and responses are $y_i = f_*({\boldsymbol x}_i)$ for $f_*$ an unknown quadratic function; (2) Feature vectors ${\boldsymbol x}_i$ are distributed as a mixture of two $…

In this paper we study the problem of learning a shallow artificial neural network that best fits a training data set. We study this problem in the over-parameterized regime where the number of observations are fewer than the number of parameters in the model. We show that with quadratic activations the optimization la…

Active-set algorithm improves Cox regression for shape-restricted covariates.

Electroencephalogram, an influential equipment for analyzing humans activities and recognition of seizure attacks can play a crucial role in designing accurate systems which can distinguish ictal seizures from regular brain alertness, since it is the first step towards accomplishing a high accuracy computer aided diagn…

Investigates the fundamental components of attention mechanisms.

Study shows overparameterization helps shallow neural networks recover signals in high dimensions.

New algorithms solve large-scale convex regression problems.

DIAL learns embeddings to maximize recall and accuracy for entity resolution.

Develops a test to distinguish between standard and rough volatility.

Optimal trading patterns adjust based on market efficiency and slippage costs.

We introduce a solution scheme for portfolio optimization problems with cardinality constraints. Typical portfolio optimization problems are extensions of the classical Markowitz mean-variance portfolio optimization model. We solve such type of problems using a method similar to column generation. In this scheme, the o…

Bayesian approach improves uncertainty in deep learning models.

One of the mysteries in the success of neural networks is randomly initialized first order methods like gradient descent can achieve zero training loss even though the objective function is non-convex and non-smooth. This paper demystifies this surprising phenomenon for two-layer fully connected ReLU activated neural n…

Wide hidden layer TCM nets capacity analyzed using RDT and fl RDT.

New MIP formulations for neural network Lipschitz constant estimation.

Study assesses neural nets for optimization problems, highlighting SiLU's effectiveness.