arXiv research

The paper analyzes the risk of bagging regularized M-estimators under proportional asymptotics.

Study ridge ensembles in proportional feature-to-sample size regime, proving risk equivalence and GCV consistency.

A new method reduces variance in SGMCMC by preferentially subsampling data.

In this paper we demonstrate that tempering Markov chain Monte Carlo samplers for Bayesian models by recursively subsampling observations without replacement can improve the performance of baseline samplers in terms of effective sample size per computation. We present two tempering by subsampling algorithms, subsampled…

Data-driven discovery of differential equations has been an emerging research topic. We propose a novel algorithm subsampling-based threshold sparse Bayesian regression (SubTSBR) to tackle high noise and outliers. The subsampling technique is used for improving the accuracy of the Bayesian learning algorithm. It has tw…

The study provides theoretical foundations for using smaller instances to predict algorithm performance on larger ones.

The paper analyzes bagging in overparameterized learning, deriving risk properties and optimal subsample sizes.

Large sample size brings the computation bottleneck for modern data analysis. Subsampling is one of efficient strategies to handle this problem. In previous studies, researchers make more fo- cus on subsampling with replacement (SSR) than on subsampling without replacement (SSWR). In this paper we investigate a kind of…

Nyström subsampling with Tikhonov regularization for covariate shift adaptation under misspecified case

New research finds many coreset methods for logistic regression are not better than simple sampling.

The paper explores effective data selection methods for weakly supervised learning.

The Sampled Gaussian Mechanism's noise level decreases with larger subsampling rates, improving privacy-utility trade-offs.

Large batch sizes reduce gradient variance in DP-SGD, improving privacy.

In this paper, we study random subsampling of Gaussian process regression, one of the simplest approximation baselines, from a theoretical perspective. Although subsampling discards a large part of training data, we show provable guarantees on the accuracy of the predictive mean/variance and its generalization ability.…

Subbagging estimation for big data reduces memory usage while maintaining statistical consistency.

A new method reduces feature screening cost from $O(np)$ to $O(\sqrt{n}p)$.

Develops a method to optimize hyperparameters for subsampling methods.

New equivalences found between subsampling and ridge regularization methods.

We describe an adaptation of the simulated annealing algorithm to nonparametric clustering and related probabilistic models. This new algorithm learns nonparametric latent structure over a growing and constantly churning subsample of training data, where the portion of data subsampled can be interpreted as the inverse …

A new algorithm PBNN improves scalability of Bayesian Neural Networks.

A genetic algorithm improves multivariate kernel density estimation.

Robust data fusion via subsampling improves model performance for rare data types.

In this paper, we study the Nystr{ö}m type subsampling for large scale kernel methods to reduce the computational complexities of big data. We discuss the multi-penalty regularization scheme based on Nystr{ö}m type subsampling which is motivated from well-studied manifold regularization schemes. We develop a theoretica…

Learnable multiclass hypothesis classes don't always have a sample compression scheme of fixed size.

We propose a simple approach which, given distributed computing resources, can nearly achieve the accuracy of $k$-NN prediction, while matching (or improving) the faster prediction time of $1$-NN. The approach consists of aggregating denoised $1$-NN predictors over a small number of distributed subsamples. We show, bot…

Develops a faster model selection method using influence functions.

In the setting of nonparametric regression, we propose and study a combination of stochastic gradient methods with Nyström subsampling, allowing multiple passes over the data and mini-batches. Generalization error bounds for the studied algorithm are provided. Particularly, optimal learning rates are derived considerin…

Random forests remain among the most popular off-the-shelf supervised learning algorithms. Despite their well-documented empirical success, however, until recently, few theoretical results were available to describe their performance and behavior. In this work we push beyond recent work on consistency and asymptotic no…

Data augmentation is commonly used to encode invariances in learning methods. However, this process is often performed in an inefficient manner, as artificial examples are created by applying a number of transformations to all points in the training set. The resulting explosion of the dataset size can be an issue in te…

We introduce extensions of stability selection, a method to stabilise variable selection methods introduced by Meinshausen and Bühlmann (J R Stat Soc 72:417-473, 2010). We propose to apply a base selection method repeatedly to random observation subsamples and covariate subsets under scrutiny, and to select covariates …

For optimization on large-scale data, exactly calculating its solution may be computationally difficulty because of the large size of the data. In this paper we consider subsampled optimization for fast approximating the exact solution. In this approach, one gets a surrogate dataset by sampling from the full data, and …

We consider stochastic second-order methods for minimizing smooth and strongly-convex functions under an interpolation condition satisfied by over-parameterized models. Under this condition, we show that the regularized subsampled Newton method (R-SSN) achieves global linear convergence with an adaptive step-size and a…

Learn2Evaluate uses learning curves to estimate high-dimensional prediction performance.

Statistical machine learning models should be evaluated and validated before putting to work. Conventional k-fold Monte Carlo Cross-Validation (MCCV) procedure uses a pseudo-random sequence to partition instances into k subsets, which usually causes subsampling bias, inflates generalization errors and jeopardizes the r…

Test partial effects in Frechet regression on Bures-Wasserstein manifolds.

A fast algorithm for $K$-means clustering using subsampled SDP.

MFRDE uses medians of forest estimators to robustly estimate densities in noisy data.

Optimizes survey design for private mean estimation with reduced variance.

State-of-the-art methods for Convolutional Sparse Coding usually employ Fourier-domain solvers in order to speed up the convolution operators. However, this approach is not without shortcomings. For example, Fourier-domain representations implicitly assume circular boundary conditions and make it hard to fully exploit …

ECV method optimizes ensemble parameters for randomized ensembles.

A new method selects a representative subsample for efficient kernel density estimation.

New algorithm samples matrix rows proportional to their ℓ_p norm in a turnstile data stream.

This paper optimizes subsampling for large datasets using Poisson distribution.

A significant hurdle for analyzing large sample data is the lack of effective statistical computing and inference methods. An emerging powerful approach for analyzing large sample data is subsampling, by which one takes a random subsample from the original full sample and uses it as a surrogate for subsequent computati…

We study the problem of subsampling in differential privacy (DP), a question that is the centerpiece behind many successful differentially private machine learning algorithms. Specifically, we provide a tight upper bound on the Rényi Differential Privacy (RDP) (Mironov, 2017) parameters for algorithms that: (1) subsamp…

A new model-free subsampling method using uniform designs is proposed.

For massive data, the family of subsampling algorithms is popular to downsize the data volume and reduce computational burden. Existing studies focus on approximating the ordinary least squares estimate in linear regression, where statistical leverage scores are often used to define subsampling probabilities. In this p…

Unified framework for subsampling mechanisms with tighter privacy guarantees.