arXiv research

Local PCA detects intrinsic parameterization of complex thermo-chemical state-spaces.

Local convergence theory for mildly over-parameterized neural nets.

In this paper, we analyze the effects of depth and width on the quality of local minima, without strong over-parameterization and simplification assumptions in the literature. Without any simplification assumption, for deep nonlinear neural networks with the squared loss, we theoretically show that the quality of local…

Twisted $\operatorname{SL}_2 \mathbb{C}$ local systems on surfaces of finite type appear often in geometry and physics. Most of them arise geometrically as local systems of charts for pleated hyperbolic structures. Bonahon and Thurston's "shear-bend coordinates" parameterize these local systems of charts. On a surface …

Policy gradient converges linearly with Hadamard parameterization in tabular settings.

Gradient EM converges globally for over-parameterized Gaussian mixtures.

We develop a more efficient NGD method for structured parameters.

Proposes a continuous, differentiable model from local adaptive models.

Over-parameterization makes optimization easier for simple neural networks, even with minor extra neurons.

We investigate a local reparameterizaton technique for greatly reducing the variance of stochastic gradients for variational Bayesian inference (SGVB) of a posterior over model parameters, while retaining parallelizability. This local reparameterization translates uncertainty about global parameters into local noise th…

The paper proposes methods for volumetric parameterization of 3D solid manifolds.

New theory for local parameterization of deep ReLU networks.

Conformal surface parameterization is useful in graphics, imaging and visualization, with applications to texture mapping, atlas construction, registration, remeshing and so on. With the increasing capability in scanning and storing data, dense 3D surface meshes are common nowadays. While meshes with higher resolution …

Develops a method for conformal parameterization of point clouds without fixed boundaries.

Locally adapted parameterizations of a model (such as locally weighted regression) are expressive but often suffer from high variance. We describe an approach for reducing the variance, based on the idea of estimating simultaneously a transformed space for the model, as well as locally adapted parameterizations in this…

Model accurately calibrates FX market skew for exotic options.

Harmonic basis vector fields on surfaces

New method tackles over-parameterized matrix sensing with FGD, improving statistical and computational complexity.

We consider the optimization problem associated with training simple ReLU neural networks of the form $\mathbf{x}\mapsto \sum_{i=1}^{k}\max\{0,\mathbf{w}_i^\top \mathbf{x}\}$ with respect to the squared loss. We provide a computer-assisted proof that even if the input distribution is standard Gaussian, even if the dime…

New algorithm calibrates local volatility from option prices using deep neural networks.

MFNs parameterize non-local interactions through matrix equivariant functions, improving graph neural network performance.

We study the lifting of the Schubert stratification of the homogeneous space of complete real flags of $R^{n+1}$ to its universal covering group $Spin_{n+1}$. We call the lifted strata the Bruhat cells of $Spin_{n+1}$, in keeping with the homonymous classical decomposition of reductive algebraic groups. We present expl…

LEAPS samples discrete distributions via CTMCs and locally equivariant networks.

Expectation Maximization (EM) is among the most popular algorithms for maximum likelihood estimation, but it is generally only guaranteed to find its stationary points of the log-likelihood objective. The goal of this article is to present theoretical and empirical evidence that over-parameterization can help EM avoid …

Unified framework for nonconvex matrix completion with linearly parameterized factors.

Does over-parameterization eliminate sub-optimal local minima for neural networks? An affirmative answer was given by a classical result in [59] for 1-hidden-layer wide neural networks. A few recent works have extended the setting to multi-layer neural networks, but none of them has proved every local minimum is global…

This work proposes a mathematical framework for loss landscapes and optimization in deep neural networks.

We identify a class of over-parameterized deep neural networks with standard activation functions and cross-entropy loss which provably have no bad local valley, in the sense that from any point in parameter space there exists a continuous path on which the cross-entropy loss is non-increasing and gets arbitrarily clos…

PrecGD restores linear convergence in over-parameterized nonconvex matrix factorization.

Improved VAE models avoid posterior collapse in text modeling.

We propose a new static parameterization of the implied volatility surface which is constructed by using polynomials of sigmoid functions combined with some other terms. This parameterization is flexible enough to fit market implied volatilities which demonstrate smile or skew. An arbitrage-free calibration algorithm i…

Proposes a new approach to generate sparse models from deep networks.

Traditionally, neural networks are parameterized using optimization procedures such as stochastic gradient descent, RMSProp and ADAM. These procedures tend to drive the parameters of the network toward a local minimum. In this article, we employ alternative "sampling" algorithms (referred to here as "thermodynamic para…

Recent work has noted that all bad local minima can be removed from neural network loss landscapes, by adding a single unit with a particular parameterization. We show that the core technique from these papers can be used to remove all bad local minima from any loss landscape, so long as the global minimum has a loss o…

We show compactness in the locally smooth topology for certain natural families of asymptotically conical self-expanding solutions of mean curvature flow. Specifically, we show such compactness for the set of all two-dimensional self-expanders of a fixed topological type and, in all dimensions, for the set of self-expa…

Constructing an efficient parameterization of a large, noisy data set of points lying close to a smooth manifold in high dimension remains a fundamental problem. One approach consists in recovering a local parameterization using the local tangent plane. Principal component analysis (PCA) is often the tool of choice, as…

This paper analyzes the landscape of supervised contrastive loss in over-parameterized networks.

Deep Reinforcement Learning (DRL) has been applied to address a variety of cooperative multi-agent problems with either discrete action spaces or continuous action spaces. However, to the best of our knowledge, no previous work has ever succeeded in applying DRL to multi-agent problems with discrete-continuous hybrid (…

Latent variable models are an elegant framework for capturing rich probabilistic dependencies in many applications. However, current approaches typically parametrize these models using conditional probability tables, and learning relies predominantly on local search heuristics such as Expectation Maximization. Using te…

Gradient descent can find better tensor decompositions than lazy training in over-parameterized settings.

This paper shows that every sublevel set of the loss function of a class of deep over-parameterized neural nets with piecewise linear activation functions is connected and unbounded. This implies that the loss has no bad local valleys and all of its global minima are connected within a unique and potentially very large…

New neural operators learn structured patterns efficiently.

The paper introduces various canonical parameterizations for 2D-curved shapes.

The paper analyzes how noise geometry influences the performance of SGD in machine learning.

There are currently two parameterizations used to derive fixed kernels corresponding to infinite width neural networks, the NTK (Neural Tangent Kernel) parameterization and the naive standard parameterization. However, the extrapolation of both of these parameterizations to infinite width is problematic. The standard p…

This paper is devoted to the application of B-splines to volatility modeling, specifically the calibration of the leverage function in stochastic local volatility models and the parameterization of an arbitrage-free implied volatility surface calibrated to sparse option data. We use an extension of classical B-splines …

The paper introduces combinatorial curvature and flow for polyhedral surfaces, proving rigidity and solving the Yamabe problem.

Combining deep learning and ensemble smoothers for better history matching.