arXiv research

This paper shows using sub-sample estimates can improve optimization results in large-scale problems.

We study covariance matrix estimation for the case of partially observed random vectors, where different samples contain different subsets of vector coordinates. Each observation is the product of the variable of interest with a $0-1$ Bernoulli random variable. We analyze an unbiased covariance estimator under this mod…

Sub-sampling is a common and often effective method to deal with the computational challenges of large datasets. However, for most statistical models, there is no well-motivated approach for drawing a non-uniform subsample. We show that the concept of an asymptotically linear estimator and the associated influence func…

Large scale optimization problems are ubiquitous in machine learning and data analysis and there is a plethora of algorithms for solving such problems. Many of these algorithms employ sub-sampling, as a way to either speed up the computations and/or to implicitly implement a form of statistical regularization. In this …

New method uses PDMPs with sub-sampling for efficient sampling from posterior distributions.

Boosts change-point detection power with optimal sub-sampling.

Bayesian realized EGARCH models improve tail risk forecasting.

Paper analyzes Nyström regularization for time series forecasting with sequential sub-sampling.

Efficient bandit exploration for various distributions without distribution-specific tuning.

We consider non-parametric estimation and inference of conditional moment models in high dimensions. We show that even when the dimension $D$ of the conditioning variable is larger than the sample size $n$, estimation and inference is feasible as long as the distribution of the conditioning variable has small intrinsic…

Many data-fitting applications require the solution of an optimization problem involving a sum of large number of functions of high dimensional parameter. Here, we consider the problem of minimizing a sum of $n$ functions over a convex constraint set $\mathcal{X} \subseteq \mathbb{R}^{p}$ where both $n$ and $p$ are lar…

The realized GARCH framework is extended to incorporate the two-sided Weibull distribution, for the purpose of volatility and tail risk forecasting in a financial time series. Further, the realized range, as a competitor for realized variance or daily returns, is employed in the realized GARCH framework. Further, sub-s…

Efficiently approximates statistical leverage scores for faster KRR.

Computing risk measures of a financial portfolio comprising thousands of derivatives is a challenging problem because (a) it involves a nested expectation requiring multiple evaluations of the loss of the financial portfolio for different risk scenarios and (b) evaluating the loss of the portfolio is expensive and the …

Acquisition of Magnetic Resonance Imaging (MRI) scans can be accelerated by under-sampling in k-space (i.e., the Fourier domain). In this paper, we consider the problem of optimizing the sub-sampling pattern in a data-driven fashion. Since the reconstruction model's performance depends on the sub-sampling pattern, we c…

In portable, three dimensional, and ultra-fast ultrasound imaging systems, there is an increasing demand for the reconstruction of high quality images from a limited number of radio-frequency (RF) measurements due to receiver (Rx) or transmit (Xmit) event sub-sampling. However, due to the presence of side lobe artifact…

Mini-batch sub-sampling in neural network training is unavoidable, due to growing data demands, memory-limited computational resources such as graphical processing units (GPUs), and the dynamics of on-line learning. In this study we specifically distinguish between static mini-batch sub-sampled loss functions, where mi…

Corrects bias in random sampling matrices for improved ML methods.

We consider the problem of minimizing a sum of $n$ functions over a convex parameter set $\mathcal{C} \subset \mathbb{R}^p$ where $n\gg p\gg 1$. In this regime, algorithms which utilize sub-sampling techniques are known to be effective. In this paper, we use sub-sampling techniques together with low-rank approximation …

Paper introduces efficient online sub-sampling for RL with function approximation, reducing policy updates.

A new realized conditional autoregressive Value-at-Risk (VaR) framework is proposed, through incorporating a measurement equation into the original quantile regression model. The framework is further extended by employing various Expected Shortfall (ES) components, to jointly estimate and forecast VaR and ES. The measu…

The standard approach to compressive sampling considers recovering an unknown deterministic signal with certain known structure, and designing the sub-sampling pattern and recovery algorithm based on the known structure. This approach requires looking for a good representation that reveals the signal structure, and sol…

This paper tackles efficient and scalable estimation of a complex model involving stochastic linear combinations of non-linear regressions.

This paper enhances stability selection by evaluating overall results robustness and identifying optimal regularization values.

BDMBC clusters data with varying densities using a new PLLS measure.

We consider the minimization of non-convex functions that typically arise in machine learning. Specifically, we focus our attention on a variant of trust region methods known as cubic regularization. This approach is particularly attractive because it escapes strict saddle points and it provides stronger convergence gu…

We use statistical learning methods to construct an adaptive state estimator for nonlinear stochastic systems. Optimal state estimation, in the form of a Kalman filter, requires knowledge of the system's process and measurement uncertainty. We propose that these uncertainties can be estimated from (conditioned on) past…

Improved statistical computation through efficient matrix sampling.

Markov chain Monte Carlo (MCMC) algorithms for hidden Markov models often rely on the forward-backward sampler. This makes them computationally slow as the length of the time series increases, motivating the development of sub-sampling-based approaches. These approximate the full posterior by using small random subsequ…

The joint Value at Risk (VaR) and expected shortfall (ES) quantile regression model of Taylor (2017) is extended via incorporating a realized measure, to drive the tail risk dynamics, as a potentially more efficient driver than daily returns. Both a maximum likelihood and an adaptive Bayesian Markov Chain Monte Carlo m…

Study sharpens threshold for matching correlated graphs without labels.

A new algorithm selects data subsets avoiding outliers and high leverage points.

For data sets populated by a very well modeled process and by another process of unknown probability density function (PDF), a desired feature when manipulating the fraction of the unknown process (either for enhancing it or suppressing it) consists in avoiding to modify the kinematic distributions of the well modeled …

We consider variants of trust-region and cubic regularization methods for non-convex optimization, in which the Hessian matrix is approximated. Under mild conditions on the inexact Hessian, and using approximate solution of the corresponding sub-problems, we provide iteration complexity to achieve $ε$-approximate seco…

StreaMRAK improves KRR for streaming data.

End-to-end analysis of SGD for STL with adaptive sub-sampling.

We consider assets for which price $X_t$ and squared volatility $Y_t$ are jointly driven by Heston joint stochastic differential equations (SDEs). When the parameters of these SDEs are estimated from $N$ sub-sampled data $(X_{nT}, Y_{nT})$, estimation errors do impact the classical option pricing PDEs. We estimate thes…

Paper proposes an efficient bandit-based algorithm for hyperparameter optimization.

We propose a randomized second-order method for optimization known as the Newton Sketch: it is based on performing an approximate Newton step using a randomly projected or sub-sampled Hessian. For self-concordant functions, we prove that the algorithm has super-linear convergence with exponentially high probability, wi…

The question of how to incorporate curvature information in stochastic approximation methods is challenging. The direct application of classical quasi- Newton updating techniques for deterministic optimization leads to noisy curvature estimates that have harmful effects on the robustness of the iteration. In this paper…

Granger causality reviewed and advanced for complex data.

We present a detailed analysis of \emph{observable} moments based parameter estimators for the Heston SDEs jointly driving the rate of returns $R_t$ and the squared volatilities $V_t$. Since volatilities are not directly observable, our parameter estimators are constructed from empirical moments of realized volatilitie…

A novel distributed adaptive NN classifier for large data sets.

The task of reconstructing a matrix given a sample of observedentries is known as the matrix completion problem. It arises ina wide range of problems, including recommender systems, collaborativefiltering, dimensionality reduction, image processing, quantum physics or multi-class classificationto name a few. Most works…

In supervised learning using kernel methods, we often encounter a large-scale finite-sum minimization over a reproducing kernel Hilbert space (RKHS). Large-scale finite-sum problems can be solved using efficient variants of Newton method, where the Hessian is approximated via sub-samples of data. In RKHS, however, the …

We consider the problem of efficiently computing the maximum likelihood estimator in Generalized Linear Models (GLMs) when the number of observations is much larger than the number of coefficients ($n \gg p \gg 1$). In this regime, optimization algorithms can immensely benefit from approximate second order information.…

Convolutional neural networks (CNNs) have achieved remarkable performance in many applications, especially in image recognition tasks. As a crucial component of CNNs, sub-sampling plays an important role for efficient training or invariance property, and max-pooling and arithmetic average-pooling are commonly used sub-…

We present novel minibatch stochastic optimization methods for empirical risk minimization problems, the methods efficiently leverage variance reduced first-order and sub-sampled higher-order information to accelerate the convergence speed. For quadratic objectives, we prove improved iteration complexity over state-of-…