arXiv research

We consider the smoothing probabilities of hidden Markov model (HMM). We show that under fairly general conditions for HMM, the exponential forgetting still holds, and the smoothing probabilities can be well approximated with the ones of double sided HMM. This makes it possible to use ergodic theorems. As an applicatio…

A neural network with a single hidden layer can't represent certain multivariable functions.

Cubic spline smoothing improves interpolation between irregularly sampled data.

Revisits the connection between neural networks and the Kolmogorov-Arnold theorem.

Improved online learning for hidden-convex losses achieves optimal regret.

New method extracts hidden phases in binary mixtures using tubular tilings.

Hidden cost: Smoothing shrinks decision boundaries, affecting class-wise accuracy.

In this paper, we consider regression problems with one-hidden-layer neural networks (1NNs). We distill some properties of activation functions that lead to $\mathit{local~strong~convexity}$ in the neighborhood of the ground-truth parameters for the 1NN squared-loss objective. Most popular nonlinear activation function…

The paper parallelizes HMM inference for efficient long-term computations.

New method shows hidden state can significantly improve differential privacy in SGD.

Study on neural networks' sample complexity with one hidden layer.

Study reveals how neural network smoothness affects their vulnerability to adversarial attacks.

Although deep learning has shown its powerful performance in many applications, the mathematical principles behind neural networks are still mysterious. In this paper, we consider the problem of learning a one-hidden-layer neural network with quadratic activations. We focus on the under-parameterized regime where the n…

Gradient Descent Ascent converges to von-Neumann solution in hidden zero-sum games.

We propose algorithms for approximate filtering and smoothing in high-dimensional Factorial hidden Markov models. The approximation involves discarding, in a principled way, likelihood factors according to a notion of locality in a factor graph associated with the emission distribution. This allows the exponential-in-d…

A new approach for functional data description is proposed in this paper. It consists of a regression model with a discrete hidden logistic process which is adapted for modeling curves with abrupt or smooth regime changes. The model parameters are estimated in a maximum likelihood framework through a dedicated Expectat…

Bayesian inference for biochemical reaction networks using jump-diffusion approximations.

Consider a feedforward neural network $ψ: \mathbb{R}^d\rightarrow \mathbb{R}^d$ such that $ψ\approx \nabla f$, where $f:\mathbb{R}^d \rightarrow \mathbb{R}$ is a smooth function, therefore $ψ$ must satisfy $\partial_j ψ_i = \partial_i ψ_j$ pointwise. We prove a theorem that a $ψ$ network with more than one hidden layer…

While neural networks are powerful approximators used to classify or embed data into lower dimensional spaces, they are often regarded as black boxes with uninterpretable features. Here we propose Graph Spectral Regularization for making hidden layers more interpretable without significantly impacting performance on th…

Proposes BATer for improved adversarial example detection.

We use smoothed analysis techniques to provide guarantees on the training loss of Multilayer Neural Networks (MNNs) at differentiable local minima. Specifically, we examine MNNs with piecewise linear activation functions, quadratic loss and a single output, under mild over-parametrization. We prove that for a MNN with …

Time series analysis is a key component of machine learning, with applications in various fields.

Optimization geometrodynamics simplifies adaptive optimizer dynamics.

Existing dimensionality reduction methods are adept at revealing hidden underlying manifolds arising from high-dimensional data and thereby producing a low-dimensional representation. However, the smoothness of the manifolds produced by classic techniques over sparse and noisy data is not guaranteed. In fact, the embed…

Spectral graph sparsification preserves geometry of GNN embeddings.

We study model recovery for data classification, where the training labels are generated from a one-hidden-layer neural network with sigmoid activations, also known as a single-layer feedforward network, and the goal is to recover the weights of the neural network. We consider two network models, the fully-connected ne…

The study proves a quantitative functional CLT for neural networks with smooth activation functions.

S-GAI initializes MLPs using spectral geometry from data, improving performance.

Smoothing graphons improve link prediction in Bayesian SBM without increasing computational complexity.

This paper introduces a new approach to finding knots and links with hidden symmetries using "hidden extensions", a class of hidden symmetries defined here. We exhibit a family of tangle complements in the ball whose boundaries have symmetries with hidden extensions, then we further extend these to hidden symmetries of…

In this paper we propose a function space approach to Representation Learning and the analysis of the representation layers in deep learning architectures. We show how to compute a weak-type Besov smoothness index that quantifies the geometry of the clustering in the feature space. This approach was already applied suc…

In this paper, we consider one dimensional (shallow) ReLU neural networks in which weights are chosen randomly and only the terminal layer is trained. First, we mathematically show that for such networks L2-regularized regression corresponds in function space to regularizing the estimate's second derivative for fairly …

New methods identify causal effects without needing complete proxy variables.

New methods for skill rating in sports using state-space models.

Exponentially smoothed RNNs improve industrial forecasting.

Gradient descent optimally trains RNNs without overparameterization.

Unified SGD method improves convergence for nested optimization problems.

SAHMM-VAE separates sources adaptively using hidden Markov priors.

Investigates optimal consumption and investment using alternative data sources.

The study examines label smoothing to improve confidence calibration in fine-tuned LLMs.

We provide new theoretical insights on why over-parametrization is effective in learning neural networks. For a $k$ hidden node shallow network with quadratic activation and $n$ training data points, we show as long as $k \ge \sqrt{2n}$, over-parametrization enables local search algorithms to find a \emph{globally} op…

RBM models reveal how hidden unit tail behavior affects pattern reconstruction.

We theoretically study the landscape of the training error for neural networks in overparameterized cases. We consider three basic methods for embedding a network into a wider one with more hidden units, and discuss whether a minimum point of the narrower network gives a minimum or saddle point of the wider one. Our re…

Expands Hidden Markov Model to include Markov chain observations.

Piecewise linear activations create many spurious local minima in neural networks.

This paper reviews recent advances in Bayesian nonparametric techniques for constructing and performing inference in infinite hidden Markov models. We focus on variants of Bayesian nonparametric hidden Markov models that enhance a posteriori state-persistence in particular. This paper also introduces a new Bayesian non…

New manifold type PNDP-manifold defined with Einstein warped product structure.

Supervised learning frequently boils down to determining hidden and bright parameters in a parameterized hypothesis space based on finite input-output samples. The hidden parameters determine the attributions of hidden predictors or the nonlinear mechanism of an estimator, while the bright parameters characterize how h…