Paper connects MoE and self-attention, proposing active-attention.
problem Improving efficiency and performance of self-attention mechanisms.
method Established connection between MoE and self-attention, analyzed quadratic gating functions, proposed active-attention mechanism.
result Active-attention outperforms standard self-attention in various tasks.
Complex-valued neural networks avoid spurious local minima.
problem Finding spurious local minima in neural networks.
method Proved no spurious local minima for shallow complex neural networks with quadratic activations.
result Complex-valued weights eliminate spurious local minima in neural networks.
Bayes-optimal learning of a neural network with quadratic activations is achieved with GAMP-RIE.
problem Learning a neural network with quadratic activations from quadratic samples.
method Combining approximate message passing with rotationally invariant matrix denoising.
result Derives a closed-form expression for Bayes-optimal test error.
Study on neural network dynamics in high dimensions with quadratic activation.
problem Understanding training dynamics in overparameterized neural networks.
method Derivation of gradient flow equations and analysis under l2-regularization.
result Characterization of estimator performance and spectral properties in the high-dimensional limit.
We show that the gradient descent algorithm provides an implicit regularization effect in the learning of over-parameterized matrix factorization models and one-hidden-layer neural networks with quadratic activations. Concretely, we show that given O ~ ( d r 2 ) \tilde{O}(dr^{2}) O ~ ( d r 2 ) random linear measurements of a rank r r r positive s…
Study on neural networks with quadratic activation functions, focusing on optimization and generalization.
problem Understanding the dynamics and generalization of neural networks with quadratic activation in the over-parametrized regime.
method Teacher-student scenario, empirical loss landscape analysis, gradient descent dynamics, numerical experiments.
result Conditions for the neural network to recover the teacher and achieve small generalization error.
The paper studies quadratic neural networks, proving existence of spurious minima and saddle points.
problem Understanding the loss landscape of neural networks with quadratic activations.
method Theoretical analysis of mean squared error loss for neural networks with quadratic activations.
result Proves existence of spurious local minima and saddle points in the training landscape of deep overparameterized quadratic neural networks.
Sharp asymptotics reveal how network width controls learnability in quadratic neural networks.
problem Understanding learnability in overparameterized quadratic neural networks.
method Mapping ERM to convex matrix sensing with nuclear norm penalization.
result Characterization of global minima and precise generalization thresholds.
Full-batch GD outperforms one-pass SGD in learning a single-index model with quadratic activation.
problem Learning a single-index model with quadratic activation using gradient descent.
method Full-batch gradient descent compared to one-pass stochastic gradient descent (SGD) on a correlation loss.
result Full-batch GD requires only n ≃ d n \simeq d n ≃ d samples for strong recovery, while one-pass SGD requires n ≳ d log d n \gtrsim d\log d n ≳ d log d samples. Study active learning of PTFs with derivative access.
problem Active learning of polynomial threshold functions (PTFs).
method Algorithm for active learning degree- d d d univariate PTFs with derivative access. result Computational efficient algorithm for active learning degree- d d d univariate PTFs. New algorithm tackles stochastic optimization with inequality constraints.
problem Stochastic optimization with inequality constraints in various applications.
method Active-set stochastic sequential quadratic programming (StoSQP) with a differentiable exact augmented Lagrangian.
result Global convergence for any initialization, KKT residuals converge to zero almost surely.
This paper extends the convergence rate of DEQs with ReLU to any general activation.
problem Proving global convergence rate for DEQs with general activations.
method Developed a novel population Gram matrix and new form of dual activation with Hermite polynomial expansion.
result Gradient descent converges to a globally optimal solution at a linear rate for DEQs with general activations.
New analysis shows capacity of treelike neural networks with various activations.
problem Analyzing the capacity of treelike neural networks with diverse activations.
method Utilized Random Duality Theory and its partially lifted version to handle various activations.
result The capacity of treelike neural networks decreases for large network width but converges to a constant value.
We provide new theoretical insights on why over-parametrization is effective in learning neural networks. For a k k k hidden node shallow network with quadratic activation and n n n training data points, we show as long as k ≥ 2 n k \ge \sqrt{2n} k ≥ 2 n , over-parametrization enables local search algorithms to find a \emph{globally} op…
Gradient descent recovers planted weights in shallow neural networks with quadratic activations.
problem Learning shallow neural networks with quadratic activations and planted weights.
method Analysis of optimization landscape, gradient descent, semicircle law for Wishart ensemble.
result Gradient descent can recover planted weights if initialized below an energy barrier.
Polynomial neural networks explore thresholds for maximum expressiveness.
problem Understanding the limits of polynomial neural networks' expressiveness.
method Introducing activation degree threshold to measure network expressiveness and proving its existence and upper bounds.
result Polynomial neural networks with equi-width architectures achieve the maximum expressiveness.
One-pass SGD dynamics in overparameterized quadratic networks show slow escape from poor solutions.
problem Slow escape from poor generalization solutions in overparameterized neural networks.
method Analysis of one-pass SGD dynamics using ordinary differential equations for overlap matrices.
result Overparameterization only modestly accelerates escape from poor solutions.
Efficiently selects nearest neighbors for labeling to speed up active learning.
problem Intractable active learning and search for large-scale unlabeled data.
method Restricts candidate pool to nearest neighbors of labeled set.
result Achieved similar performance to global approach but reduced computational cost by up to 3 orders of magnitude.
Gradient descent dynamics in quadratic regression models are analyzed, revealing five phases: monotonic, catapult, periodic, chaotic, and divergent.
problem Analyzing the dynamics of gradient descent in quadratic regression models.
method Fine-grained bifurcation analysis of gradient descent dynamics using a cubic map parameterized by the step-size.
result Gradient descent dynamics in quadratic regression models exhibit five distinct phases: monotonic, catapult, periodic, chaotic, and divergent.
The problem of active diagnosis arises in several applications such as disease diagnosis, and fault diagnosis in computer networks, where the goal is to rapidly identify the binary states of a set of objects (e.g., faulty or working) by sequentially selecting, and observing, (noisy) responses to binary valued queries. …
We describe a framework for designing efficient active learning algorithms that are tolerant to random classification noise and are differentially-private. The framework is based on active learning algorithms that are statistical in the sense that they rely on estimates of expectations of functions of filtered random e…
Deep neural networks enforce non-crossing quantile regression curves.
problem Estimating quantile regression curves without crossing.
method Penalized deep ReQU neural networks with a non-crossing penalty.
result Established non-asymptotic risk and error bounds for the estimated QRP.
Catapult phase in neural nets shows exponential loss growth before quick decrease.
problem Understanding phase transitions in neural networks during training.
method Analyzing weight norm and loss behavior for super-critical learning rates.
result Proven existence of catapult phase in quadratic models and two-layer nets.
Improved stability analysis of neural network systems using Zames-Falb multipliers.
problem Analyzing stability of linear systems with neural network nonlinearities.
method Using integral quadratic constraints, sector-bounded and slope-restricted structure, and acausal Zames-Falb multipliers.
result Flexible and versatile framework for stability analysis with improved computational efficiency.
Finding minimum distortion of adversarial examples and thus certifying robustness in neural network classifiers for given data points is known to be a challenging problem. Nevertheless, recently it has been shown to be possible to give a non-trivial certified lower bound of minimum adversarial distortion, and some rece…
We show LLMs can be locally linear, enabling better control of activations.
problem Suboptimal control of LLM activations during generation.
method Model LLM inference as a linear dynamical system, compute feedback controllers using Jacobians, and adapt classical control theory.
result Robust, fine-grained control of LLM activations across models and tasks.
New SDP method certifies neural network robustness across all classes efficiently.
problem Certifying robustness of neural networks across multiple classes.
method Quadratic model + SDP relaxation + pruning strategy.
result Significant computational speed-up and scalability to large datasets.
Over-parametrization speeds up learning a single neuron model.
problem Understanding why over-parametrization accelerates learning in neural networks.
method Studied a simple model of a single teacher neuron with quadratic activation, showing how over-parametrization can lead to faster convergence.
result Over-parametrization helps gradient descent enter the neighborhood of a global optimal solution faster.
New method speeds up neural kernel computations for various activations.
problem Inefficient computation of neural kernels for general activations.
method Fast sketching method using truncated Hermite expansion.
result 106x speedup for approximate CNTK computation on CIFAR-10.
New lower bounds improve logistic log-likelihood optimization and inference.
problem Designing computationally tractable lower bounds for logistic log-likelihoods.
method Developed a piece-wise quadratic lower bound that uniformly improves tangent quadratic minorizers.
result Improves the speed of convergence and accuracy of variational Bayes approximations.
Traditional human activity recognition (HAR) based on time series adopts sliding window analysis method. This method faces the multi-class window problem which mistakenly labels different classes of sampling points within a window as a class. In this paper, a HAR algorithm based on U-Net is proposed to perform activity…
New method improves neural network robustness by identifying functions rather than parameters.
problem Neural networks' lack of robustness to distribution shifts.
method Identify the function represented by quadratic networks, not their parameters.
result Obtain robust generalization bounds for neural networks.
Study on SGD dynamics and scaling laws for training quadratic neural networks in high dimensions.
problem Optimizing and understanding the training dynamics of quadratic neural networks in high-dimensional settings.
method Sharp analysis of SGD dynamics, combining matrix Riccati differential equations and matrix monotonicity arguments.
result Derivation of scaling laws for prediction risk, highlighting power-law dependencies on optimization time, sample size, and model width.
We study the supervised learning problem under either of the following two models: (1) Feature vectors x i {\boldsymbol x}_i x i are d d d -dimensional Gaussians and responses are y i = f ∗ ( x i ) y_i = f_*({\boldsymbol x}_i) y i = f ∗ ( x i ) for f ∗ f_* f ∗ an unknown quadratic function; (2) Feature vectors x i {\boldsymbol x}_i x i are distributed as a mixture of two $…
In this paper we study the problem of learning a shallow artificial neural network that best fits a training data set. We study this problem in the over-parameterized regime where the number of observations are fewer than the number of parameters in the model. We show that with quadratic activations the optimization la…
Active-set algorithm improves Cox regression for shape-restricted covariates.
problem Improving Cox regression for shape-restricted covariates.
method Shape-restricted inference using active-set optimization for spline basis expansion.
result Active-set algorithm produces accurate linear covariate effect estimates.
Electroencephalogram, an influential equipment for analyzing humans activities and recognition of seizure attacks can play a crucial role in designing accurate systems which can distinguish ictal seizures from regular brain alertness, since it is the first step towards accomplishing a high accuracy computer aided diagn…
Investigates the fundamental components of attention mechanisms.
problem Understanding the building blocks of attention in deep learning.
method Classified and studied three key mechanisms: additive, multiplicative output, and synaptic attention.
result Additive activation attention is central in proofs of lower bounds.
Study shows overparameterization helps shallow neural networks recover signals in high dimensions.
problem Signal recovery in shallow neural networks with overparameterization.
method Gradient flow on population risk, Gaussian distribution assumption, high-dimensional limit analysis.
result Minimal overparameterization is sufficient for strong recovery of signals.
New algorithms solve large-scale convex regression problems.
problem Large-scale convex regression with subgradient regularization.
method Active set type algorithm on dual QP, approximate optimization, randomized augmentation.
result Solves problems with n=10^5 and d=10 in minutes.
DIAL learns embeddings to maximize recall and accuracy for entity resolution.
problem Low resource settings for entity resolution with large Cartesian product search space.
method DIAL uses an Index-By-Committee framework with pre-trained transformer language models to jointly learn embeddings for recall and accuracy.
result DIAL achieves high precision, recall, and efficiency on benchmark datasets.
Develops a test to distinguish between standard and rough volatility.
problem Determining whether asset volatility follows a standard semimartingale or a rough process.
method Uses sample autocovariance of high-frequency asset return data to detect negative autocorrelation at high frequencies.
result Evidence of rough volatility in SPY high-frequency data.
Optimal trading patterns adjust based on market efficiency and slippage costs.
problem Balancing active alphas and trading costs in active portfolios.
method Maximization of utility including projected alpha-based profits, slippage costs, and risk aversion.
result Optimal trading involves a no-trade zone width that scales as Δ ∼ c 1 / 2 Δ\sim c^{1/2} Δ ∼ c 1/2 , differing from stochastic settings. We introduce a solution scheme for portfolio optimization problems with cardinality constraints. Typical portfolio optimization problems are extensions of the classical Markowitz mean-variance portfolio optimization model. We solve such type of problems using a method similar to column generation. In this scheme, the o…
Bayesian approach improves uncertainty in deep learning models.
problem Uncertainty quantification in deep learning models.
method Bayesian point of view, Gaussian approximability, semi-parametric Bernstein-von Mises theorems.
result Bayesian credible regions have valid frequentist coverage, providing theoretical justification for deep learning.
One of the mysteries in the success of neural networks is randomly initialized first order methods like gradient descent can achieve zero training loss even though the objective function is non-convex and non-smooth. This paper demystifies this surprising phenomenon for two-layer fully connected ReLU activated neural n…
Wide hidden layer TCM nets capacity analyzed using RDT and fl RDT.
problem Capacity analysis of wide hidden layer TCM nets.
method Employed Fully Lifted Random Duality Theory (fl RDT) for capacity characterization.
result Explicit, closed form capacity characterizations for a generic class of hidden layer activations.
New MIP formulations for neural network Lipschitz constant estimation.
problem Ensuring robustness of neural networks by calculating their Lipschitz constant.
method Reformulating the neural network Lipschitz estimation problem as a Quadratically Constrained MIP (MIQCQP) problem.
result Solutions of the MIQCQP formulations provide bounds on the Lipschitz constant, with conditions for exactness.