New guarantees for SGD in non-convex optimization without strict noise bounds.
problem Efficiently escaping saddle points in non-convex optimization.
method Mean-square arguments and relaxed gradient noise variance bounds.
result Gradient descent can efficiently escape saddle points with a more relaxed gradient noise variance bound.
Comparing with traditional learning criteria, such as mean square error (MSE), the minimum error entropy (MEE) criterion is superior in nonlinear and non-Gaussian signal processing and machine learning. The argument of the logarithm in Renyis entropy estimator, called information potential (IP), is a popular MEE cost i…
New results show flat minima in neural networks suffer from high dimensionality.
problem Flat minima in neural networks generalize poorly in high dimensions.
method Theoretical analysis of two-layer ReLU networks with multivariate inputs.
result Flat minima lead to exponentially slower convergence in high dimensions.
We propose two sparsity-aware normalized subband adaptive filter (NSAF) algorithms by using the gradient descent method to minimize a combination of the original NSAF cost function and the l1-norm penalty function on the filter coefficients. This l1-norm penalty exploits the sparsity of a system in the coefficients upd…
Double Q-learning has the same mean-squared error as Q-learning under certain conditions.
problem Comparing the mean-squared error of Double Q-learning and Q-learning.
method Theoretical analysis based on Lyapunov equations for both tabular and linear function approximation settings.
result The asymptotic mean-squared error of Double Q-learning is exactly equal to that of Q-learning under specific conditions.
The kernel least mean squares (KLMS) algorithm is a computationally efficient nonlinear adaptive filtering method that "kernelizes" the celebrated (linear) least mean squares algorithm. We demonstrate that the least mean squares algorithm is closely related to the Kalman filtering, and thus, the KLMS can be interpreted…
Estimates latent inner products from an anisotropic Gaussian graph with improved spectral method.
problem Recovering latent inner products from an anisotropic Gaussian random geometric graph.
method Doubly centered adjacency matrix, rank-d spectral approximation, Hermite expansion, decoupling argument.
result Estimator achieves mean squared error rate matching state of the art for isotropic case and ill-conditioned covariance matrices.
The paper provides mean-square error bounds for stochastic approximation algorithms.
problem Error bounds for recursive equations with Markovian disturbances.
method Analysis of mean-square error for stochastic approximation algorithms.
result Mean-square error achieves the optimal rate of O(1/n) under certain conditions. Constrained adaptive filtering algorithms inculding constrained least mean square (CLMS), constrained affine projection (CAP) and constrained recursive least squares (CRLS) have been extensively studied in many applications. Most existing constrained adaptive filtering algorithms are developed under mean square error (…
This paper analyzes sampling from heavy-tailed distributions using discretized Itô diffusions.
problem Sampling from heavy-tailed distributions with finite variance.
method Mean-square analysis of discretized Itô diffusions with weighted Poincaré inequalities.
result Explicit iteration complexity for obtaining samples close to target distributions in Wasserstein-2 metric.
This work analyzes Q-learning with adaptive stepsizes for finite-time convergence.
problem Finite-time convergence analysis for average-reward Q-learning with adaptive stepsizes. method Adaptive stepsizes as local clocks, time-inhomogeneous Markovian reformulation, almost-sure time-varying bounds, conditioning arguments, and Markov chain concentration inequalities.
result Convergence rates of ildeO(1/k) for mean-square and pointwise mean-square convergence. This paper presents a stochastic behavior analysis of a kernel-based stochastic restricted-gradient descent method. The restricted gradient gives a steepest ascent direction within the so-called dictionary subspace. The analysis provides the transient and steady state performance in the mean squared error criterion. It…
This study calculates the maximum error of a famous estimation method.
problem Estimating rare items not seen in a sample.
method Characterizes the maximal mean-squared error of the Good-Turing estimator.
result Characterizes the maximal mean-squared error of the Good-Turing estimator.
New CH covariance class improves spatial statistics by balancing differentiability and tail behavior.
problem Lack of control over mean-square differentiability and tail behavior in Matérn covariance functions.
method Developed a new Confluent Hypergeometric (CH) covariance class using a scale mixture of Matérn and polynomial covariances.
result The CH class offers improved theoretical properties and better performance in extrapolative settings.
We consider adaptive system identification problems with convex constraints and propose a family of regularized Least-Mean-Square (LMS) algorithms. We show that with a properly selected regularization parameter the regularized LMS provably dominates its conventional counterpart in terms of mean square deviations. We es…
TACAM improves argument mining by integrating topic and external context.
problem Mining arguments from text without topic information leads to confusion.
method Proposes models that consider topic information and integrate external context.
result Performance boost for argument mining when topic and external context are considered.
New method optimizes tail dependence coefficient estimation.
problem Estimating tail dependence in nonparametric data.
method Optimal threshold selection combining mean squared error and copula estimation.
result Improved accuracy in tail dependence coefficient estimation.
Nonparametric modeling approaches show very promising results in the area of system identification and control. A naturally provided model confidence is highly relevant for system-theoretical considerations to provide guarantees for application scenarios. Gaussian process regression represents one approach which provid…
Cryptocurrency prices predicted using LSTM, SVM, and polynomial regression.
problem Uncertainty in crypto coin values.
method Long Short Term Memory, Support Vector Machine, Polynomial Regression models.
result Support Vector Machine with linear kernel had the smallest mean square error.
We formulate the problem of neural network optimization as Bayesian filtering, where the observations are the backpropagated gradients. While neural network optimization has previously been studied using natural gradient methods which are closely related to Bayesian inference, they were unable to recover standard optim…
New method finds best arm minimizing mean-squared error in correlated bandits.
problem Finding an arm that best captures information about other arms.
method Formulated correlated bandit problem, derived MSE estimator, proposed algorithm.
result Proposed algorithm identifies best arm with error probability bounds.
Study on LMMSE estimation with model mismatch, quantifying MSE trade-offs.
problem Model mismatch in LMMSE estimation with undermodeling.
method Analyzing the average MSE of LMMSE estimation with random regressors.
result Performance improvement depends on sufficient samples and model complexity.
We define two new notions of projection of a stochastic differential equation (SDE) onto a submanifold: the Ito-vector and Ito-jet projections. This allows one to systematically develop low dimensional approximations to high dimensional SDEs using differential geometric techniques. The approach generalizes the notion o…
A fast method for LOOCV in k-NN regression reduces computation time.
problem Efficient computation of LOOCV for k-NN regression.
method Identical LOOCV estimate to (k+1)-NN MSE on training data.
result LOOCV computation can be done with (k+1)-NN regression once.
Develops optimal low-dimensional approximations to high-dimensional SDEs.
problem Approximating solutions to high-dimensional SDEs in a low-dimensional space.
method Introduces Ito-vector and Ito-jet projections for optimal approximation.
result Optimal projection filters yield better approximations than Stratonovich projection.
Kernel adaptive filters (KAF) are a class of powerful nonlinear filters developed in Reproducing Kernel Hilbert Space (RKHS). The Gaussian kernel is usually the default kernel in KAF algorithms, but selecting the proper kernel size (bandwidth) is still an open important issue especially for learning with small sample s…
New methods for estimating complex causal effects in econometrics.
problem Estimating causal parameters in short panel data models using nested nonparametric instrumental variable regression.
method Introducing techniques to limit ill-posedness in nested NPIV, providing explicit mean square rates and efficient inference.
result Explicit mean square rates for nested NPIV and efficient inference for causal parameters.
We study the relationship between online Gaussian process (GP) regression and kernel least mean squares (KLMS) algorithms. While the latter have no capacity of storing the entire posterior distribution during online learning, we discover that their operation corresponds to the assumption of a fixed posterior covariance…
Unified framework for finite-sample RL algorithms using Lyapunov theory.
problem Finite-sample convergence guarantees of asynchronous RL algorithms.
method Reformulate RL algorithms as Markovian SA, develop Lyapunov analysis.
result Mean-square error bounds and convergence for various RL algorithms.
The purpose of this paper is to indicate that the recently proposed Momentum fractional least mean squares (mFLMS) algorithm has some serious flaws in its design and analysis. Our apprehensions are based on the evidence we found in the derivation and analysis in the paper titled: \textquotedblleft \textit{Momentum frac…
Paper presents a Siamese network for identifying more convincing evidence.
problem Identifying the more convincing argument in discussions.
method Proposes a Siamese neural network architecture for labeling evidence as more convincing.
result The Siamese network outperforms baselines on a new challenging data set.
The most important aspect of any classifier is its error rate, because this quantifies its predictive capacity. Thus, the accuracy of error estimation is critical. Error estimation is problematic in small-sample classifier design because the error must be estimated using the same data from which the classifier has been…
This letter presents an improved version of diffusion least mean ppower (LMP) algorithm for distributed estimation. Instead of sum of mean square errors, a weighted sum of mean square error is defined as the cost function for global and local cost functions of a network of sensors. The weight coefficients are updated b…
Paper explores arbitrage and CAPM in continuous time.
problem Understanding arbitrage and CAPM in continuous time.
method Analyzes instantaneous arbitrage and its relation to CAPM.
result Arbitrage and CAPM arguments differ in assumptions about the market portfolio.
Enhances RL for jump processes using MSBVE algorithm.
problem Challenges in continuous-time RL with jumps and noise.
method Introduces MSBVE algorithm to minimize quadratic variation error.
result MSBVE algorithm outperforms MSTDE in jump processes.
Paper optimizes diffusion models for denoising tasks with theoretical guarantees.
problem Lack of theoretical understanding of MSE optimality in diffusion models.
method Inspired by MSE-optimal CME, proposes a novel denoising strategy for diffusion models.
result Demonstrates polynomial-time convergence to the CME under mild conditions.
Improved multi-task averaging reduces mean squared error in high-dimensional data.
problem Joint estimation of multiple distributions using independent data sets.
method Exploits similarities between tasks by shrinking naive estimators towards local averages.
result The method provides a significant reduction in mean squared error, especially in high-dimensional spaces.
ABae efficiently computes subset means with expensive predicates using stratified sampling.
problem Computing subset means with expensive predicates efficiently.
method Stratified sampling and proxy models.
result Mean squared error of O(N−1) when N is split evenly between stages. Simplified argument for second order estimate in quaternionic Calabi-Yau problem.
problem Second order estimates for quaternionic Calabi-Yau problem on hyperkähler manifolds.
method Simplified argument to derive the estimate.
result Simplified derivation of second order estimate.
Study non-asymptotic estimation bounds for LTI models with Gaussian noise.
problem Estimating parameters of LTI models with non-asymptotic error bounds.
method Sharp non-asymptotic lower bounds using Cramér-Rao and van Trees inequalities, concentration results, and differential geometric constructions.
result Sharp and rate-optimal lower bounds for mean square estimation risk.
This is a continuation of our first paper in [WY16]. There are two purposes of this paper: One is to give a proof of the main result in [WY16] without going through the argument depending on numerical effectiveness. The other one is to provide a proof of our conjecture, mentioned in [TY], where the assumption of negati…
MIC improves VAR order selection accuracy.
problem Order selection in VAR models for accurate forecasting.
method MIC based on expected squared error loss.
result MIC consistently estimates true VAR order.
RMSNorm simplifies LayerNorm, reducing computational cost.
problem Computational overhead of LayerNorm in deep neural networks.
method RMSNorm re-scales inputs using root mean square (RMS) instead of invariance.
result RMSNorm achieves comparable performance to LayerNorm but reduces running time.
Unified framework for robust A/B testing under model misspecification.
problem Improving sample efficiency in A/B testing with model misspecification.
method Unified framework for contextual bandit and dynamic settings, proving worst-case mean squared error bounds.
result Empirically validated approach using synthetic and real-world datasets.
Improved stock volume prediction using Kalman Filters with various hidden states.
problem Improving accuracy of intraday trading volume prediction.
method Extended Kalman Filter with various hidden states for different stocks, using cross-validation to determine optimal state number.
result Demonstrated improved accuracy through comparison experiments and numerical analysis.
For J-holomorphic mappings for a strongly pseudo-convex manifold, we prove elliptic regularity by the argument of boots-strapping.
Transformer model with mixed-frequency data improves stock volatility prediction.
problem Improving stock volatility prediction using mixed-frequency data.
method Transformer model trained on mixed-frequency data (GARCH-MIDAS model for frequency alignment).
result Transformer model reduces mean square error from 1.00 to 0.86.
Estimating mean from one-bit samples of symmetric log-concave distributions.
problem Estimating the mean of a symmetric log-concave distribution with limited one-bit measurements.
method Analyzes mean squared error in three settings: centralized, adaptive, and distributed, with and without quantization.
result One round of adaptivity is sufficient to achieve optimal mean-square error in the adaptive setting.