A new method reduces variance in SGMCMC by preferentially subsampling data.
problem High variance in stochastic gradient estimates impacts sampler performance.
method Use a non-uniform probability distribution to preferentially subsample data points and adaptively adjust subsample size.
result Maintains accuracy while substantially reducing average subsample size.
SGLDiff approximates Bayesian posterior distributions with subsampling error.
problem Approximating Bayesian posterior distributions in large-scale data settings.
method Stochastic Gradient Langevin Diffusion (SGLDiff) with subsampling.
result The Wasserstein distance between the posterior and SGLDiff's limiting distribution is bounded by a fractional power of the mean waiting time.
Efficient methods for training deep neural networks using subsampled Gauss-Newton and natural gradient.
problem Training deep neural networks with large datasets and variables.
method Subsampled Gauss-Newton and natural gradient methods with subsampled gradient estimates.
result Methods converge to a stationary point and are efficient to implement.
The paper analyzes Newton-like and inexact Newton methods for stochastic optimization problems.
problem Optimization of stochastic problems with approximated gradients and Hessians.
method Newton-like methods using subsampled gradients and Hessians, and inexact Newton methods using conjugate gradient for Hessian approximation.
result Inexact Newton methods can achieve similar convergence rates to exact methods, with lower computational cost.
Optimal rates found for learning with Nyström stochastic gradient methods.
problem Nonparametric regression learning with improved computational efficiency.
method Combination of stochastic gradient methods with Nyström subsampling, allowing multiple passes and mini-batches.
result Derivation of optimal learning rates considering various parameters.
Improved sampling accuracy in SG-MCMC methods via non-uniform gradient subsampling.
problem Computational inefficiency and sampling error in stochastic gradient MCMC methods.
method Proposes a non-uniform subsampling scheme to reduce sampling error in EWSG, a variant of SG-MCMC.
result EWSG reduces sampling error compared to uniform subsampling, improving accuracy without sacrificing convergence speed.
A new stochastic solver improves Convolutional Sparse Coding efficiency.
problem Efficiency and sparsity in Convolutional Sparse Coding.
method Randomized subsampling strategy in spatial domain for online learning.
result Improved execution time with no loss in learning quality.
The Sampled Gaussian Mechanism's noise level decreases with larger subsampling rates, improving privacy-utility trade-offs.
problem Improving privacy-utility trade-offs in differentially private stochastic optimization.
method Proof of a conjecture about the Sampled Gaussian Mechanism's noise level and subsampling rate relationship.
result A rigorous proof of the conjecture, completing the proof of Theorem 6.2 in the original paper.
Stochastic second-order methods converge fast under interpolation conditions.
problem Minimizing smooth and strongly-convex functions efficiently.
method Regularized subsampled Newton method (R-SSN) and stochastic BFGS algorithms.
result R-SSN achieves global linear convergence and quadratic rate in a local neighbourhood.
Paper bridges statistical inference for DP-SGD, a privacy-preserving machine learning method.
problem Asymptotic statistical inference for Differentially Private Stochastic Gradient Descent (DP-SGD).
method Established asymptotic properties of SGD under randomized subsampling, extended to DP-SGD, proposed methods for constructing valid confidence intervals.
result Valid confidence intervals for DP-SGD output achieve nominal coverage rates while maintaining privacy.
Enhances financial optimization under model uncertainty using subsampling.
problem Model uncertainty in financial decision-making from limited data.
method Superimposes uncertainty measure on model space, uses subsampling for model distribution approximation, adapts SGD for efficiency.
result Uncertainty measures outperform traditional methods and achieve comparable performance to Bayesian methods.
Study on Newton-Sketch and Subsampled Newton methods for large-scale optimization.
problem Optimization of large-scale finite-sum problems with high-dimensional data.
method Hessian subsampling and randomized Hadamard transformations for dimensionality reduction in Newton's method.
result Advantages of conjugate gradient vs. stochastic gradient iterations revealed through experiments.
Enhanced privacy, utility, and efficiency through MUST subsampling.
problem Balancing privacy, utility, and computational efficiency in data analysis.
method MUltistage Sampling Technique (MUST) for privacy amplification in differential privacy.
result MUST offers stronger privacy guarantees ( ϵ \epsilon ϵ ) than one-stage subsampling methods while maintaining similar utility and computational efficiency. A new neural subsampling method reduces data volume for deep models.
problem Efficiently process huge volumes of high-dimensional data like images.
method Two-stage end-to-end neural subsampling model that optimizes for arbitrary downstream tasks.
result Outperforms baselines under low subsampling rates on various tasks.
A new algorithm speeds up machine learning by solving large-scale problems more efficiently.
problem Efficiently solving large-scale machine learning problems with regularization.
method Subsampled proximal Newton-type method that leverages finite sum structure and recent stochastic first-order methods.
result The method achieves faster convergence than state-of-the-art methods for non-smooth regularizers.
A new approach RA improves stochastic optimization by executing multiple steps between subsample updates.
problem Improving the efficiency and effectiveness of stochastic optimization methods.
method Developed Retrospective Approximation (RA) which executes multiple steps between subsample updates using a deterministic solver.
result RA achieves almost sure and L 1 L_1 L 1 consistency under weak conditions and optimizes iteration and oracle complexity. SGMCMC uses data subsampling to make MCMC scalable for large datasets.
problem Scalability of MCMC for large datasets.
method Data subsampling techniques to reduce computational cost.
result SGMCMC algorithms are more efficient than standard MCMC on benchmark examples.
Develops tests for Markowitz stochastic dominance spanning using saddle points.
problem Determining if adding securities or relaxing investment constraints improves investment opportunity sets.
method Derives properties of cdfs, defines Markowitz stochastic dominance spanning, constructs non-parametric tests based on subsampling.
result Rejects market portfolio Markowitz efficiency and finds evidence of outperformance.
We unify subsampling methods for network embeddings and prove their asymptotic distribution.
problem Understanding and improving the performance of network embeddings learned via subsampling.
method Unified framework for node2vec-like methods, proving asymptotic distribution under exchangeable graph assumption.
result Asymptotic distribution of learned embedding vectors decouples and provides rates of convergence.
Stochastic Stein Discrepancies improve inference efficiency.
problem Intractable computation of Stein discrepancies.
method Subsampled approximations of Stein operators.
result Stochastic Stein Discrepancies inherit convergence properties of standard SDs.
A new method speeds up community detection in graphs.
problem Efficiently detecting communities in large graphs.
method Subsampled semidefinite programming for faster clustering.
result Statistical limits of sketching for community detection established.
This paper explains CART random forests using stochastic control theory.
problem Understanding the inner workings of CART random forests.
method Developed a stochastic-control perspective on CART random forests, interpreting feature subsampling as a random feasible action set and the split rule as a policy.
result Established that the CART policy is locally stabilizing but globally suboptimal for the forest objective.
Proposes STRON method for large-scale machine learning problems.
problem Large-scale machine learning problems.
method Stochastic Trust Region Inexact Newton (STRON) method using CG to solve trust region subproblem with progressive subsampling.
result Empirical results show efficacy of STRON method.
New method for Bayesian inference on large datasets.
problem Scalable sampling for Bayesian generalized linear mixed models on large datasets.
method Mirror Langevin dynamics with data subsampling, post-processing for variance estimation.
result Asymptotic, order-wise correct estimation of posterior variance.
Algorithm for fast matrix factorization of large datasets.
problem Factorizing huge matrices with sparse or dense factors.
method Subsampling and iterative learning of matrix factors.
result Significant speed-ups on large datasets.
SSTQ improves privacy-preserving vector quantization with low communication cost.
problem Achieving local differential privacy in distributed optimization with low communication cost.
method Combines overcomplete equal-norm tight frames, coordinate subsampling, and privacy-aware one-dimensional quantization.
result Achieves optimal mean squared error scaling with only ⌈ log 2 N ⌉ + b \lceil \log_2 N \rceil + b ⌈ log 2 N ⌉ + b bits per client. Large batch sizes reduce gradient variance in DP-SGD, improving privacy.
problem Understanding why large batch sizes work in DP-SGD.
method Decomposed total gradient variance into subsampling and noise-induced variances, proving batch size independence in the limit.
result Large batch sizes reduce effective total gradient variance, improving privacy in DP-SGD.
Paper develops methods for statistical inference in SGD with infinite variance.
problem Challenges in statistical inference for SGD with infinite variance.
method Model-agnostic methodology based on weak convergence and subsampling calibration.
result Asymptotically valid confidence regions for SGD in both finite and infinite variance regimes.
Study exact community recovery in noisy SBM with limited queries.
problem Community recovery in noisy stochastic block models with limited queries.
method Balanced uniform querying, two-stage adaptive strategy, sublinear queries, subsampled graph.
result Adaptive querying can improve exact recovery limits in noisy SBM.
SVRN accelerates Newton methods by reducing variance and improving performance.
problem Improving the efficiency of Newton methods for large-scale optimization problems.
method Stochastic Variance-Reduced Newton (SVRN) algorithm that accelerates Subsampled Newton and Iterative Hessian Sketch algorithms.
result SVRN accelerates Newton methods by reducing the number of passes over the data, achieving a significant improvement in performance.
Structured subsampling improves privacy in deep time series forecasting.
problem Incompatible privacy guarantees with time series forecasting.
method Structured subsampling of sequential data for privacy amplification.
result Structured subsampling enables training with strong privacy guarantees.
New analysis shows SNG's effectiveness in small samples.
problem Limited understanding of SNG in small data settings.
method Sketch-and-project analysis of SNG.
result Global convergence and rate characterization for SNG.
The paper studies how adding an ℓ2 penalty affects network embeddings.
problem The impact of ℓ2 regularization on network embeddings.
method Analyzes the asymptotic behavior of ℓ2 regularized node2vec embeddings under graphon theory.
result The learned embeddings asymptotically form a graphon with a nuclear-norm-type penalty.
Boulevard boosts trees with regularization, converging to a normal distribution.
problem Improving regression model accuracy and uncertainty quantification.
method Regularized stochastic gradient boosting trees with subsampling and shrinkage.
result The algorithm converges to a normal distribution, enabling uncertainty characterization.
FALKON efficiently processes large datasets using kernel methods.
problem Limited applicability of kernel methods in large scale scenarios.
method Combining stochastic subsampling, iterative solvers, and preconditioning.
result Optimal statistical accuracy achieved with O ( n ) O(n) O ( n ) memory and O ( n n ) O(n\sqrt{n}) O ( n n ) time. Study tightens bounds on subsampled differential privacy.
problem Improving differential privacy in subsampled datasets.
method Analytical moments accounting for subsampled RDP mechanisms.
result Tight upper bound on RDP parameters for subsampled mechanisms.
A new method selects a representative subsample for efficient kernel density estimation.
problem Selecting a representative subsample without model assumptions.
method Optimal transport techniques for model-free subsampling with an efficient algorithm.
result The selected subsample can be used for efficient density estimation with derived convergence rates and optimal bandwidth.
In this paper we demonstrate that tempering Markov chain Monte Carlo samplers for Bayesian models by recursively subsampling observations without replacement can improve the performance of baseline samplers in terms of effective sample size per computation. We present two tempering by subsampling algorithms, subsampled…
SGHMC uses noise to find global minima in non-convex learning.
problem Finding global minima in non-convex optimization problems.
method SGHMC with momentum and Gaussian noise for non-asymptotic convergence.
result Non-asymptotic convergence analysis for non-convex optimization.
This paper optimizes subsampling for large datasets using Poisson distribution.
problem Efficiently subsample large datasets for quasi-likelihood estimation.
method Derives optimal Poisson subsampling probabilities and develops a distributed subsampling framework.
result Consistent and asymptotically normal estimators are obtained.
Large sample size brings the computation bottleneck for modern data analysis. Subsampling is one of efficient strategies to handle this problem. In previous studies, researchers make more fo- cus on subsampling with replacement (SSR) than on subsampling without replacement (SSWR). In this paper we investigate a kind of…
Dynamic Structural Causal Models handle time-dependent systems with cycles and latent confounding.
problem Representing and analyzing systems of Stochastic Differential Equations (SDEs) with DSCMs.
method Define time-splitting and subsampling operations to analyze DSCMs of SDEs, and apply existing causal discovery algorithms to time-series data.
result DSCMs provide a graphical Markov property for SDEs and enable identification of time-dependent causal effects.
A significant hurdle for analyzing large sample data is the lack of effective statistical computing and inference methods. An emerging powerful approach for analyzing large sample data is subsampling, by which one takes a random subsample from the original full sample and uses it as a surrogate for subsequent computati…
Proposes efficient subsampling for logistic regression with optimal probabilities.
problem Efficiently approximating maximum likelihood estimate in logistic regression for large datasets.
method Develops subsampling algorithms for logistic regression, derives optimal subsampling probabilities, and proposes two-step approximation.
result Optimal subsampling reduces computing time significantly while maintaining estimator consistency and normality.
Study detects P-type bifurcations in single system realizations using unreliable kernel density estimates.
problem Detecting P-type bifurcations in signals with unreliable kernel density estimates.
method Create persistence diagrams from single system realization, statistically analyze resulting set, compare point process modeling methods.
result Subsampling outperforms other point process modeling methods in predicting P-type bifurcations.
SMH kernel speeds up Bayesian inference for large datasets.
problem Inefficient MCMC methods for large datasets.
method Combines factorized acceptance probabilities, fast simulation, and control variates.
result Achieves geometric ergodicity and exact posterior sampling.
A new model-free subsampling method using uniform designs is proposed.
problem Model-based subsampling methods are often dependent on model assumptions.
method Developed a criterion (GEFD) and a model-free subsampling method based on uniform designs.
result The proposed method outperforms random sampling and is robust under diverse model specifications.
Doubly SGD improves convergence for intractable objective optimization.
problem Optimizing objectives in sum of intractable expectations.
method Doubly SGD with doubly stochastic gradients and independent minibatching.
result Established convergence of doubly SGD under general conditions, including dependent component gradient estimators.