Study shows how bias in optimization affects robustness in adversarial settings.
problem Understanding and mitigating implicit bias in adversarially robust models.
method Analyzes the implicit bias in robust empirical risk minimization and its impact on generalization.
result Implicit bias in optimization can significantly affect robust generalization.
Bias correction improves language model training performance.
problem Stochastic update bias in preconditioned optimizers.
method Cross-fitted preconditioning and variance-corrected inversion.
result Reduces held-out pretraining loss by 0.15 nats.
A new bias score method optimizes fairness in classification.
problem Ensuring fairness in binary classification under group constraints.
method Introducing bias scores and developing a post-hoc approach to adapt to fairness constraints.
result The method maintains high accuracy while ensuring fairness constraints.
Paper addresses underestimation bias in double Q-learning, proposing a method to improve learning performance.
problem Underestimation bias in double Q-learning leading to non-optimal fixed points.
method Proposes a simple approach using approximate dynamic programming to bound the target value.
result Significant improvement in learning performance over baseline algorithms in Atari benchmark tasks.
Develops tools to audit ML models for bias and unfairness.
problem Auditing ML models for individual bias and unfairness.
method Formalizes the task as an optimization problem and develops inferential tools for the optimal value.
result Demonstrates the utility of tools in revealing biases in COMPAS recidivism prediction instrument.
The paper analyzes optimal implicit bias in linear regression for over-parameterized models.
problem Finding the best generalization performance in over-parameterized linear regression.
method Asymptotic analysis of generalization performance for convex functions/potentials.
result Optimal implicit bias that achieves the best generalization error under certain conditions.
Improves decision-making by correcting feature selection bias.
problem Cofounding bias in feature selection affects machine learning predictions.
method Proposes a meta-algorithm using a novel adjustment criterion based on causal sufficiency.
result Corrects cofounding bias to improve prediction performance.
Optimal feature transfer identified through bias-variance analysis.
problem Optimizing feature transfer in transfer learning.
method Simple linear model with fine-grained bias-variance decomposition.
result Optimal pretrained feature transform is naturally sparse.
New insights into bias mitigation show DRO isn't a complete solution.
problem Bias in machine learning systems across different data subsets.
method Theoretical analysis of Distributionally Robust Optimization (DRO) and data curation.
result Neither DRO nor data curation alone can fully address bias issues.
Unified framework for intersectionally fair AI models using MIO.
problem Bias in AI models for high-risk domains.
method Mixed-Integer Optimization (MIO) for fairness and interpretability.
result Improved performance in detecting and mitigating bias at intersections.
Bayesian imputation optimizes bias-variance tradeoff in time-series data.
problem Look-ahead bias in imputation of missing time-series data.
method Wasserstein interpolation for Bayesian posterior consensus distribution.
result Optimal control of look-ahead bias and variance in imputation.
This work investigates implicit bias in multiclass separable data using a novel geometry-aware optimizer.
problem Understanding implicit bias in overparameterized models on multiclass separable data.
method Introduces NucGD, a geometry-aware optimizer enforcing low-rank structures through nuclear norm constraints.
result NucGD enables scalable training and characterizes the impact of stochastic optimization dynamics.
New insights into bias-variance tradeoff for data-driven optimization under local misspecification.
problem Understanding the relative performance of SAA, IEO, and ETO under local misspecification.
method Developed a local misspecification perspective using contiguity theory in statistics.
result Explicit expressions for decision bias and geometric understanding of variance.
EBQL reduces bias in Q-learning for improved performance.
problem Over- and under-estimation biases in Q-learning degrade performance.
method Ensemble Bootstrapping to reduce both over- and under-estimation biases.
result EBQL outperforms other Q-learning methods in Atari games.
New algorithm achieves optimal regret in average reward MDPs without prior bias information.
problem Achieving optimal regret in average reward MDPs with computational efficiency and without prior bias information.
method Projective Mitigated Extended Value Iteration (PMEVI) to compute bias-constrained optimal policies efficiently.
result First tractable algorithm with minimax optimal regret of O ~ ( s p ( h ∗ ) S A T ) \widetilde{\mathrm{O}}(\sqrt{\mathrm{sp}(h^*) S A T}) O ( sp ( h ∗ ) S A T ) . In classification problems, sampling bias between training data and testing data is critical to the ranking performance of classification scores. Such bias can be both unintentionally introduced by data collection and intentionally introduced by the algorithm, such as under-sampling or weighting techniques applied to i…
New methods reduce bias in estimating optimality gaps for risk-averse stochastic programs.
problem Optimality gap estimation bias in risk-averse stochastic programs.
method Two independent samples, each estimating a different component of the optimality gap.
result Our method reduces bias in estimating optimality gaps for risk-averse problems.
Study measures gender bias in machine translation using multiple reference points.
problem Measuring and identifying gender bias in machine translation.
method Used an optimal non-biased translator, reference points from occupational statistics and survey.
result Found bias against both genders, but more against women, and found occupations have a greater effect than adjectives.
New bounds show SGD can match deterministic gradient descent's convergence rate.
problem Optimizing SGD convergence rate under strong convexity and smoothness.
method Computer-aided Lyapunov analysis, focusing on bias-optimal bounds.
result SGD achieves optimal convergence rate in bias terms for a wide range of step-sizes.
Bayesian imputation optimizes bias-variance trade-off in time-series data.
problem Look-ahead bias in imputation of missing time-series data.
method Bayesian consensus posterior that fuses multiple posteriors to optimize bias and variance trade-off.
result Benefit of imputation for portfolio allocation with missing returns demonstrated.
Careful tuning of the learning rate, or even schedules thereof, can be crucial to effective neural net training. There has been much recent interest in gradient-based meta-optimization, where one tunes hyperparameters, or even learns an optimizer, in order to minimize the expected loss when the training procedure is un…
Machine learning models predict brain age with systematic bias, corrected in this study.
problem Systematic bias in machine learning regression models for brain age prediction.
method General constrained optimization approach to correct bias.
result Our method effectively eliminates the bias from brain age predictions.
Outlier based Robust Principal Component Analysis (RPCA) requires centering of the non-outliers. We show a "bias trick" that automatically centers these non-outliers. Using this bias trick we obtain the first RPCA algorithm that is optimal with respect to centering.
This work overcomes bias in concave multi-objective reinforcement learning.
problem Gradient bias in policy gradient methods for concave scalarized multi-objective reinforcement learning.
method Developed a Natural Policy Gradient (NPG) algorithm with a multi-level Monte Carlo (MLMC) estimator.
result Achieved optimal O ~ ( ε − 2 ) \widetilde{\mathcal{O}}(ε^{-2}) O ( ε − 2 ) sample complexity for computing an ε ε ε -optimal policy. A new method corrects bias in high-dimensional ridge regression.
problem Inherent bias in ridge regression limits statistical efficiency and scalability.
method Iterative bias correction strategy for p < n p < n p < n and Ridge-Screening method for p > n p > n p > n . result Valid inferences and asymptotic properties established for de-biased ridge estimators.
A new algorithm CAP learns optimal policies from observational data with confounding bias and missing observations.
problem Offline contextual bandit with confounding bias and missing observations.
method CAP policy learning, forming reward function as solution of integral equation system, building confidence set, and greedily taking action with pessimism.
result Developed an upper bound to the suboptimality of CAP for the offline contextual bandit problem.
Study shows momentum-based optimizers like Muon and MomentumGD bias towards KKT points in smooth homogeneous models.
problem Understanding the implicit bias of momentum-based optimizers on smooth homogeneous models.
method Analysis of Muon, MomentumGD, Signum, and Adam optimizers under decaying learning rate schedules.
result Momentum-based optimizers approximate steepest descent trajectories and bias towards KKT points of margin maximization problems.
DoubleGen addresses bias in generative modeling of counterfactuals.
problem Bias in generative models for counterfactual outcomes.
method Doubly robust framework that modifies generative modeling training objectives to mitigate confounding and misspecification biases.
result Successfully addresses confounding bias even if only one auxiliary model is correct.
Corrects bias in random sampling matrices for improved ML methods.
problem Inversion bias in random sampling matrices hampers ML applications.
method Corrects inversion bias for various random sampling methods.
result Establishes local convergence rates for sub-sampled Newton methods.
Self-attention optimizers converge to optimal weights, revealing bias patterns.
problem Understanding the implicit bias in self-attention mechanisms.
method Analysis of gradient-based optimization in self-attention layers.
result Adaptive step-size strategies can accelerate convergence to optimal weights.
We find the optimal error for a constrained regression model under a linear model.
problem Minimizing error while adhering to demographic parity constraints.
method Proposed a minimax optimal error analysis for a demographic parity-constrained regression problem within a linear model.
result The minimax optimal error is characterized by $Θ( rac{dM}{n})$ .
A number of applications (e.g., AI bot tournaments, sports, peer grading, crowdsourcing) use pairwise comparison data and the Bradley-Terry-Luce (BTL) model to evaluate a given collection of items (e.g., bots, teams, students, search results). Past work has shown that under the BTL model, the widely-used maximum-likeli…
New findings show DNC is not optimal for deep models, revealing a low-rank bias.
problem Theoretical limitations of DNC in non-linear models and multi-class classification.
method Analysis of non-linear models of arbitrary depth in multi-class classification.
result DNC stops being optimal for DUFM when going beyond two layers or two classes, due to a low-rank bias.
Paper tackles bias-variance trade-off in missing data, proposing a dynamic framework.
problem Missing data in practical applications deteriorates model performance.
method Develops a fine-grained dynamic learning framework to jointly optimize bias and variance.
result Theoretical and empirical validation of joint bias-variance optimization.
Study shows SGD's generalization is not explained by implicit bias.
problem Explaining the generalization ability of overparameterized learning algorithms.
method Revisited Stochastic Convex Optimization with SGD, demonstrating limitations of implicit bias.
result No distribution-independent or distribution-dependent implicit regularizer can explain SGD's generalization.
Graph neural networks improve combinatorial optimization by leveraging inductive bias.
problem Combinatorial optimization problems often arise from related data distributions.
method Using graph neural networks to enhance or solve combinatorial tasks.
result Graph neural networks effectively encode combinatorial and relational input.
Scalable algorithm for computing Wasserstein-2 barycenters without bias.
problem Computing Wasserstein-2 barycenters efficiently and accurately.
method Input convex neural networks and cycle-consistency regularization.
result Our approach avoids introducing bias and does not require minimax optimization.
Gradient descent struggles with learning a single neuron with bias.
problem Learning a single neuron with a bias term in the realizable setting with ReLU activation.
method Theoretical study using gradient descent, characterizing critical points, and providing convergence guarantees.
result Gradient descent faces significant challenges in learning a single neuron with bias, unlike the bias-less case.
Adam optimizes linear classifiers with separable data.
problem Understanding Adam's implicit bias in linear logistic regression.
method Study of Adam's behavior on linearly separable data.
result Adam converges to a linear classifier with maximum ℓ ∞ \ell_\infty ℓ ∞ -margin. ARFF reduces spectral bias in SGD-trained neural networks.
problem Spectral bias in two-layer neural networks.
method Comparison of SGD and ARFF on spectral bias and robustness.
result ARFF yields a closer to zero spectral bias compared to SGD.
Paper explores robust regression methods and their bias-variance trade-off.
problem Understanding the trade-off between robust estimation and optimization methods.
method Examines traditional outlier-resistant robust estimation and robust optimization.
result Both methods follow converse strategies due to a bias-variance trade-off.
We study the implicit bias of generic optimization methods, such as mirror descent, natural gradient descent, and steepest descent with respect to different potentials and norms, when optimizing underdetermined linear regression or separable linear classification problems. We explore the question of whether the specifi…
Fair Adversarial Networks remove bias from data.
problem Bias in datasets leads to biased outcomes, making analysis illegal and sub-optimal.
method Remove bias from data by altering all proxy variables, ensuring fairness without changing analytical pipelines.
result Fair Adversarial Networks effectively remove bias from data.
Adam avoids simplicity bias in neural networks, leading to better generalization.
problem Simplicity bias in neural networks trained with SGD.
method Comparison of Adam and GD on binary classification tasks with Gaussian data.
result Adam leads to richer and more diverse features, improving generalization.
The study addresses biases in evaluating molecular optimization methods and proposes methods to reduce these biases.
problem Biases in in silico evaluation of molecular optimization methods.
method Discussion and empirical investigation of bias reduction methods for predictor misspecification and sample reuse.
result Empirical investigation of bias reduction methods for predictor misspecification and sample reuse.
In many applications we seek to maximize an expectation with respect to a distribution over discrete variables. Estimating gradients of such objectives with respect to the distribution parameters is a challenging problem. We analyze existing solutions including finite-difference (FD) estimators and continuous relaxatio…
Stochastic Gradient Descent shows directional bias with moderate learning rates, impacting optimization outcomes.
problem Understanding the bias of SGD with moderate learning rates in practical scenarios.
method Analyzing SGD and GD on an overparameterized linear regression problem.
result SGD converges along large eigenvalue directions, GD along small ones, affecting early stopping outcomes.
The paper corrects bias in fluid approximation for better decision-making in stochastic optimization.
problem Bias introduced by using mean values in fluid approximation leads to suboptimal decisions.
method Identifying a decision-corrected point estimate that yields optimal decisions.
result A corrected point estimate exists under certain conditions and can be computed algorithmically.