We study differentially private (DP) algorithms for stochastic convex optimization (SCO). In this problem the goal is to approximately minimize the population loss given i.i.d. samples from a distribution over convex and Lipschitz loss functions. A long line of existing work on private convex optimization focuses on th…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,738 papers · 148 categories
Trend · papers per month
48 results for “Population Loss Minimizer”
Though machine learning algorithms excel at minimizing the average loss over a population, this might lead to large discrepancies between the losses across groups within the population. To capture this inequality, we introduce and study a notion we call maximum weighted loss discrepancy (MWLD), the maximum (weighted) d…
Training activation quantized neural networks involves minimizing a piecewise constant function whose gradient vanishes almost everywhere, which is undesirable for the standard back-propagation or chain rule. An empirical way around this issue is to use a straight-through estimator (STE) (Bengio et al., 2013) in the ba…
Convergence of continuous-time stochastic gradient descent with applications to deep neural networkscs.LG
Continuous-time SGD converges under certain conditions, useful for deep learning.
problem Minimizing population expected loss in learning problems.
method Continuous-time approximation of stochastic gradient descent.
result Establishes sufficient conditions for convergence, applicable to overparametrized neural networks.
Study merges datasets to improve AI model performance.
problem Improving machine learning models with diverse datasets.
method Developed an algorithm using oracle inequality and data-driven estimators.
result Algorithm reduces population loss with high probability.
The classical asymptotic theory for parametric -estimators guarantees that, in the limit of infinite sample size, the excess risk has a chi-square type distribution, even in the misspecified case. We demonstrate how self-concordance of the loss allows to characterize the critical sample size sufficient to guarantee …
Study connects covariance cleaning theory to information theory for heavy-tailed distributions.
problem Optimizing covariance matrices for heavy-tailed distributions using information theory.
method Minimizing Frobenius norm and information loss between true and estimated covariance matrices.
result Asymptotic regime of large matrices minimizes information loss for Student's t distributions.
Paper analyzes agnostic learning of mixed linear regression without generative models.
problem Learning mixed linear regression without assuming stochastic generation.
method Expectation Maximization (EM) and Alternating Minimization (AM) algorithms.
result AM and EM algorithms converge to population loss minimizers under standard conditions.
New framework for DP-SMO with near-optimal privacy-loss trade-off.
problem Optimal trade-off between privacy and population loss in DP-SMO.
method General framework using Phased-ERM method and black-box optimization.
result Near-linear time algorithms with near-optimal guarantees.
We study the problem of finding the best linear model that can minimize least-squares loss given a data-set. While this problem is trivial in the low dimensional regime, it becomes more interesting in high dimensions where the population minimizer is assumed to lie on a manifold such as sparse vectors. We propose proje…
We analyze double descent in finite-width neural networks using influence functions.
problem Understanding double descent in finite-width neural networks.
method Using influence functions to derive population loss bounds and investigate loss function effects.
result Derived bounds exhibit double descent behavior at the interpolation threshold.
In stochastic optimization, the population risk is generally approximated by the empirical risk. However, in the large-scale setting, minimization of the empirical risk may be computationally restrictive. In this paper, we design an efficient algorithm to approximate the population risk minimizer in generalized linear …
New algorithm for ML models in gradually adapting data settings.
problem Training models when data distribution reacts to the model over time.
method Stateful Performative Gradient Descent (Stateful PerfGD)
result Stateful PerfGD minimizes performative loss in gradually adapting data settings.
Uncertainty sampling, a popular active learning algorithm, is used to reduce the amount of data required to learn a classifier, but it has been observed in practice to converge to different parameters depending on the initialization and sometimes to even better parameters than standard training on all the data. In this…
A clustering method for multivariate populations with similar dependence structures.
problem Grouping populations with similar dependence structures.
method Orthogonal projection coefficients of density copulas estimated from populations.
result Clusters of populations with similar dependence structures.
Most high-dimensional estimation and prediction methods propose to minimize a cost function (empirical risk) that is written as a sum of losses associated to each data point. In this paper we focus on the case of non-convex losses, which is practically important but still poorly understood. Classical empirical process …
Polyak step size GD reaches final radius of convergence after log iterations.
problem Statistical and computational complexities of Polyak step size GD.
method Generalized smoothness and Lojasiewicz conditions, stability of gradients.
result Polyak step size GD reaches final statistical radius of convergence after logarithmic number of iterations.
Optimal private ERM and SCO with subquadratic gradient complexity.
problem Private optimization of non-smooth convex functions.
method Subquadratic gradient complexity algorithm using subsampling and smoothing.
result Achieved optimal excess empirical risk and population loss.
Paper simplifies balancing weights by relaxing outcome assumptions.
problem Estimating missing outcomes in a target population.
method Relaxes outcome assumptions to simplify balancing weights.
result Balancing weights can be simplified with convex loss and minimum worst-case bias.
We study the Stochastic Gradient Langevin Dynamics (SGLD) algorithm for non-convex optimization. The algorithm performs stochastic gradient descent, where in each step it injects appropriately scaled Gaussian noise to the update. We analyze the algorithm's hitting time to an arbitrary subset of the parameter space. Two…
We solve a high-dimensional model where nonlinear autoencoders detect hidden structure missed by PCA.
problem Hidden structure in high-dimensional data not detected by PCA.
method Tractable spiked model with two latent factors, one visible and one uncorrelated.
result Nonlinear autoencoders can extract hidden structure missed by PCA, even if reconstruction loss is higher.
New DP algorithm improves privacy and efficiency for convex optimization.
problem Efficient, DP algorithms for convex optimization with strong excess risk bounds.
method Output perturbation for a broad class of tilted loss functions.
result Near optimal DP excess risk and runtime bounds for convex optimization.
Neural networks are vulnerable to adversarial examples and researchers have proposed many heuristic attack and defense mechanisms. We address this problem through the principled lens of distributionally robust optimization, which guarantees performance under adversarial input perturbations. By considering a Lagrangian …
A new analytical approach to consistency and overfitting in regularized empirical risk minimizationmath.ST
This work considers the problem of binary classification: given training data from a certain population, together with associated labels , determine the best label for an element not among the training data. More specifically, this work considers a variant o…
New algorithms achieve optimal DP convex optimization with linear time and gradient computations.
problem Private stochastic convex optimization with optimal excess loss.
method Two new techniques: variable batch sizes and localization with stable optimization.
result Achieves optimal bound on excess loss with gradient computations.
Improved privacy bounds for learning linear predictors with convex losses.
problem Differentially private learning of linear predictors with convex losses.
method Developed private model selection approach to achieve optimal rates.
result Improved upper and lower bounds for excess population risk.
ScoreStop uses gradient tests to stop gradient boosting early.
problem Overfitting in gradient boosted decision trees.
method ScoreStop uses a functional score test based on gradients to stop boosting.
result ScoreStop is competitive with loss-based early stopping methods.
We address the problem of correcting group discriminations within a score function, while minimizing the individual error. Each group is described by a probability density function on the set of profiles. We first solve the problem analytically in the case of two populations, with a uniform bonus-malus on the zones whe…
Proposes -balancing for more balanced expert utilization in MoE models.
problem Balanced expert utilization in MoE models to avoid bias.
method Directly targets population-level balance by minimizing a convex potential function.
result Consistently outperforms prior methods in stability and effectiveness.
This guide simplifies high-probability regret bounds in empirical risk minimization.
problem High-probability regret bounds in empirical risk minimization.
method Modular presentation, three-step recipe, localized Rademacher complexity, local maximal inequalities, metric-entropy integrals.
result Recover familiar rates for various function classes and derive regret bounds for nuisance components.
Gradient EM converges exponentially to optimal solution in agnostic mixtures.
problem Fitting parametric functions to given data points without a generative model.
method Gradient EM algorithm for agnostic mixtures of arbitrary parametric functions.
result Gradient EM converges exponentially to population loss minimizers with high probability.
Study reveals sharp characterisation of local minima in neural network loss landscapes.
problem Characterizing local minima in high-dimensional two-layer ReLU neural networks.
method Exact low-dimensional representation of local minima using summary statistics and link with one-pass SGD dynamics.
result Local minima in overparameterized neural networks form discrete families with varying stability and reachability.
Gradient descent learns over-param neural nets better than NTK.
problem Learning over-parametrized neural networks with ReLU activations.
method Gradient descent from random initialization on a Gaussian input distribution.
result Gradient descent achieves population loss , while NTK achieves .
The paper provides a method to find optimal machine learning model parameters with confidence.
problem Finding optimal machine learning model parameters that generalize well to the entire population.
method Constructs valid confidence sets for the optimal parameter using only training data.
result Valid confidence sets for optimal machine learning model parameters can be generated using bootstrapping techniques.
New method learns population dynamics from snapshots, outperforming existing models.
problem Capturing periodic and other dynamical properties of population dynamics.
method Wasserstein Lagrangian Mechanics (WLM) for learning second-order dynamics from observed marginals.
result WLM outperforms existing methods across various dynamics, including vortex dynamics, embryonic development, and flocking.
A new active learning method considers both uncertainty and diversity to minimize labeling and decision costs.
problem Classical AL approaches fail to capture data distribution in unlabeled data, leading to mislabeling of outliers.
method CBAL considers classification uncertainty and instance diversity, using a min-max approach to minimize labeling and decision costs.
result Extensive experiments show CBAL outperforms state-of-the-art AL approaches.
IRM fails to capture natural invariances on simple problems.
problem IRM fails to capture natural invariances on simple problems.
method IRM formulation and practical linear variant.
result IRM can lead to worse generalization than unconstrained ERM.
Unified framework for automatic debiased machine learning for various statistical parameters.
problem Inference on smooth functionals of nonparametric M-estimands.
method Unified framework using gradient, Hessian, and linear approximation; solves two risk minimization problems.
result Efficient autoDML estimators with double robustness and robustness to misspecification.
The landscape of empirical risk has been widely studied in a series of machine learning problems, including low-rank matrix factorization, matrix sensing, matrix completion, and phase retrieval. In this work, we focus on the situation where the corresponding population risk is a degenerate non-convex loss function, nam…
New algorithms for differentially private optimization in convex and non-convex settings with near-optimal rates.
problem Differentially private optimization in convex and non-convex settings.
method Developed algorithms for convex and non-convex settings with near-optimal excess population risk.
result Achieved near-optimal rates in near-linear time for convex settings and nearly dimension independent rates for non-convex settings.
The paper analyzes the maximum margin algorithm's performance on noisy data.
problem Analyzing the performance of maximum margin algorithm on noisy data.
method Finite-sample analysis of maximum margin algorithm applied to noisy data.
result The maximum margin algorithm can achieve nearly optimal population risk with sufficient over-parameterization.
The study optimizes bounds for comparing training and population loss.
problem Optimizing bounds for comparing training and population loss.
method Derives generic information-theoretic and PAC-Bayesian generalization bounds using convex comparator functions.
result The tightest possible bound is obtained with the comparator being the convex conjugate of the CGF of the bounding distribution.
Private Stochastic Non-Convex Optimization: Adaptive Algorithms and Tighter Generalization Boundscs.LG
Improved DP algorithms for non-convex optimization with tighter generalization bounds.
problem Private stochastic non-convex optimization in high-dimensional spaces.
method Differential privacy techniques, including adaptive algorithms like DP RMSProp and DP Adam, combined with adaptive data analysis.
result Achieved a sharper rate of for population loss, improving upon previous bounds.
Optimal convex loss function improves regression coefficient estimation.
problem Asymptotic variance improvement in linear regression estimation.
method Score matching extension for log-concave projection.
result Semiparametric estimator attains minimal asymptotic covariance.
Paper improves privacy-preserving optimization rates for convex functions.
problem Differentially private stochastic convex optimization.
method Algorithmic improvements for convex and strongly convex functions under TNC and non-negative loss.
result Excess population risk bounds for DP-SCO are faster than previous results.
Gradient descent learns a single neuron without knowing the relationship between inputs and labels.
problem Learning a single neuron without knowing the relationship between inputs and labels.
method Using gradient descent to minimize empirical risk over i.i.d. samples, with a nonconvex and nonsmooth optimization problem.
result Gradient descent achieves near-optimal population risk in polynomial time and sample complexity.
Subspace Perspective on Canonical Correlation Analysis: Dimension Reduction and Minimax Ratesmath.ST
Canonical correlation analysis (CCA) is a fundamental statistical tool for exploring the correlation structure between two sets of random variables. In this paper, motivated by recent success of applying CCA to learn low dimensional representations of high dimensional objects, we propose to quantify the estimation loss…
Method leverages population data to deconvolve unknown noise and model parameters.
problem Deconvolution of unknown observational noise in distributional inversion problems.
method Large data sets from physical systems, modified gradient descent, active learning.
result Simultaneous deconvolution of noise and model parameters.