Two simple methods learn fair metrics from data to improve fairness in ML tasks.
problem Lack of widely accepted fair metrics for many ML tasks hinders individual fairness adoption.
method Presented two simple ways to learn fair metrics from various data types.
result Fair training with learned metrics improves fairness on three ML tasks.
Proposes a method to select fair performance metrics through metric elicitation.
problem Choosing fair performance metrics in multiclass classification with multiple sensitive groups.
method Metric elicitation strategy that requires only relative preference feedback and is robust to noise.
result Elicits group-fair performance metrics for multiclass classification problems.
Revises individual fairness by finding a fair metric for a model.
problem Difficulties in specifying a suitable fairness metric a priori.
method Introduces minimal metrics and applies randomized smoothing from adversarial robustness.
result Adapting minimal metrics to complex models yields interpretable fairness guarantees.
The paper clusters group fairness metrics and visualizes them using PCA.
problem Measuring discrimination against socio-demographic groups using multiple metrics.
method Empirical observation and PCA visualization of fairness metrics.
result Group fairness metrics cluster into two or three main clusters.
Algorithm learns similarity metrics for individual fairness.
problem Difficulty in learning similarity metrics for individual fairness.
method Gradient descent and Bradley-Terry model for pairwise comparisons.
result Algorithm converges to ground truth metric for individual fairness.
We study fairness in collaborative-filtering recommender systems, which are sensitive to discrimination that exists in historical data. Biased data can lead collaborative-filtering methods to make unfair predictions for users from minority groups. We identify the insufficiency of existing fairness metrics and propose f…
The paper connects counterfactual fairness to robust prediction and group fairness using causal context.
problem The challenge of ensuring fairness in AI systems when counterfactuals cannot be directly observed.
method Using causal context to bridge counterfactual fairness, robust prediction, and group fairness.
result Counterfactual fairness is equivalent to group fairness metrics in specific contexts.
BADR framework optimizes fairness metrics efficiently.
problem Fairness-inefficient models in machine learning.
method Bilevel Adaptive Rescalarisation procedure.
result BADR framework recovers optimal Pareto-efficient models.
UDJ-FL framework achieves multiple distributive justice-based fairness metrics in federated learning.
problem Ensuring fairness in federated learning across different client data distributions.
method UDJ-FL framework uses aleatoric uncertainty-based client weighing and fair resource allocation techniques.
result UDJ-FL achieves egalitarian, utilitarian, Rawls' difference principle, and desert-based fairness metrics.
This paper develops a new method for eliciting more flexible metrics, improving fairness and applicability.
problem Limited flexibility in existing metric elicitation strategies for reflecting user preferences.
method Develops a strategy for eliciting quadratic metrics based on predictive rates, requiring only relative preference feedback.
result Achieves near-optimal query complexity and broadens the use cases for metric elicitation.
Unified approach for fair classification with overlapping groups.
problem Ensuring fairness across multiple overlapping groups in prediction problems.
method Probabilistic population analysis leading to Bayes-optimal classifier, unifying existing methods.
result Outperforms baselines in fairness-performance tradeoff on real datasets.
Algorithm identifies intended fairness constraints from expert demonstrations for fair clustering.
problem Fair clustering challenges due to incomplete fairness constraints.
method Algorithm identifies fairness metric from expert demonstrations and generates clusters.
result Algorithm identifies and generates fair clusters from limited expert demonstrations.
Extracts fairness truth from classifiers using an oracle.
problem Fairness in classification algorithms without context.
method Uses fairness oracle to learn underlying fairness truth.
result Extracts metric fairness from classifiers.
New clustering method considers causal fairness to avoid bias.
problem Clustering algorithms can unintentionally propagate unfair disparities.
method Integrates causal fairness metrics into clustering algorithms.
result Demonstrates efficacy on datasets with known unfair biases.
This work shows how evaluation metrics can be seen as fair gambles.
problem The relationship and evaluation of machine learning forecasts.
method Using game-theoretic probability, the authors show evaluation metrics as fair gambles.
result Standard evaluation metrics are fair gambler outcomes, with calibration and regret metrics on two dimensions.
Developing classification algorithms that are fair with respect to sensitive attributes of the data has become an important problem due to the growing deployment of classification algorithms in various social contexts. Several recent works have focused on fairness with respect to a specific metric, modeled the correspo…
The paper explores fairness metrics in automated decision-making and their limitations.
problem Discrimination in automated resource allocation decisions.
method Analysis of fairness metrics and distributive justice principles.
result Prominent fairness metrics fail to address egalitarian and sufficiency concerns in resource allocation.
A new fairness metric for decision-making algorithms, conditioning on known fair variables.
problem Fairness issues in decision-making systems.
method Conditional fairness metric, Derivable Conditional Fairness Regularizer (DCFR), adversarial representation.
result Traditional fairness notations are special cases of the new conditional fairness notation.
Extends Demographic Parity for fairer wage predictions with expert knowledge.
problem Inadequate fairness metrics limit user domain knowledge and ignore intersectional fairness.
method Develops a parametric method to incorporate expert knowledge in fair predictions.
result Offers a robust solution for real-life applications with limited data and spending constraints.
Recommender systems are one of the most pervasive applications of machine learning in industry, with many services using them to match users to products or information. As such it is important to ask: what are the possible fairness risks, how can we quantify them, and how should we address them? In this paper we offer …
We consider the problem of improving fairness when one lacks access to a dataset labeled with protected groups, making it difficult to take advantage of strategies that can improve fairness but require protected group labels, either at training or runtime. To address this, we investigate improving fairness metrics for …
We identify and optimize the fairness-accuracy tradeoff through TAF Curves and FAUC metrics.
problem Balancing fairness and accuracy in machine learning models for high-stakes decisions.
method Developed TAF Curves and FAUC metric to quantify the tradeoff, and introduced FairStacks framework to expand the Pareto frontier.
result FairStacks framework expands the empirical Pareto frontier and improves the FAUC for model ensembles.
There has been much discussion recently about how fairness should be measured or enforced in classification. Individual Fairness [Dwork, Hardt, Pitassi, Reingold, Zemel, 2012], which requires that similar individuals be treated similarly, is a highly appealing definition as it gives strong guarantees on treatment of in…
As more researchers have become aware of and passionate about algorithmic fairness, there has been an explosion in papers laying out new metrics, suggesting algorithms to address issues, and calling attention to issues in existing applications of machine learning. This research has greatly expanded our understanding of…
Recent work in fairness in machine learning has proposed adjusting for fairness by equalizing accuracy metrics across groups and has also studied how datasets affected by historical prejudices may lead to unfair decision policies. We connect these lines of work and study the residual unfairness that arises when a fairn…
This paper evaluates fairness in deep metric learning and proposes a method to reduce subgroup performance gaps.
problem The negative impact of deep metric learning representations on minority subgroup performance in downstream tasks.
method Definition of fairness in DML through inter-class, intra-class, and uniformity properties; finDML benchmark; Partial Attribute De-correlation (PARADE) method.
result Bias in DML representations propagates to downstream tasks, even with balanced training data.
This paper improves fairness in recommendation systems by learning individual preferences across multiple dimensions.
problem Fairness in recommender systems, especially in areas with social impact.
method Opportunistic multi-aspect re-ranking approach that learns individual preferences and enhances provider fairness.
result Achieves a better trade-off between accuracy and fairness across multiple fairness dimensions.
Machine fairness is impossible to achieve fully due to historical biases.
problem Machine learning models inherit biases from historical data, making it impossible to satisfy fairness metrics simultaneously.
method Presented a causal perspective to the impossibility theorem of fairness.
result It is impossible to satisfy fairness metrics like demographic parity, equal opportunity, and equalized odds simultaneously.
Fair GLASSO estimates fair GGMs by balancing statistical dependencies across groups.
problem Fairness in graphical models with biased data.
method Regularized graphical lasso with bias metrics, proximal gradient algorithm.
result Preserves statistical accuracy while promoting fairness across groups.
Develops a fair post-processing method for student success predictions.
problem Ensuring fairness in predictive student models for educational applications.
method Uses the MADD metric to improve model fairness while maintaining accuracy.
result Successfully improved fairness of predictive models for student success.
We revisit the notion of individual fairness proposed by Dwork et al. A central challenge in operationalizing their approach is the difficulty in eliciting a human specification of a similarity metric. In this paper, we propose an operationalization of individual fairness that does not rely on a human specification of …
New metric MADD assesses fairness of predictive student models.
problem Predictive student models can be biased and unfair, leading to discrimination.
method Proposes MADD metric to analyze model's discriminatory behaviors.
result Fair predictive performance does not guarantee fair behaviors or outcomes.
New LFR algorithm ensures fair predictions with theoretical guarantees.
problem Ensuring fairness in AI algorithms for social decision-making.
method Proposes a new adversarial training scheme using IPM with a parametric family of discriminators.
result Theoretical guarantee of fairness in final prediction models.
Fairness-aware classification is receiving increasing attention in the machine learning fields. Recently research proposes to formulate the fairness-aware classification as constrained optimization problems. However, several limitations exist in previous works due to the lack of a theoretical framework for guiding the …
The paper assesses fairness in risk score models, focusing on epistemic value.
problem Fairness of risk score models in communicating uncertainty.
method Identified key fairness desiderata, developed metrics for quantitative assessment, and applied methodology in two case studies.
result Introduced a novel calibration error metric for meaningful comparisons between groups of different sizes.
New algorithm tackles subgroup fairness in AI with multiple sensitive attributes.
problem Heavy computational burdens and data sparsity in subgroup fairness for multiple sensitive attributes.
method Doubly Regressing Adversarial learning (DRAF) for subgroup fairness, focusing on subgroups with sufficient sample sizes and marginal fairness.
result DRAF algorithm reduces a surrogate fairness gap for supIPM with less computation than directly reducing supIPM.
CAT framework improves AI medical screening fairness and reliability.
problem Imbalanced data, varying performance across cohorts, and patient-level inconsistencies in traditional metrics.
method CAT framework introduces patient-level assessment, entropy-based distribution weighting, and cohort-weighted sensitivity and specificity.
result Enhanced predictive reliability, fairness, and interpretability of AI-driven medical screening models.
Two new methods assess feature importance for fairness in machine learning models.
problem Understanding how features influence fairness in machine learning models.
method Two model-agnostic approaches: permutation and occlusion.
result Simple, scalable, and interpretable methods to quantify feature importance for fairness.
SenSeI ensures fair models by enforcing invariance on sensitive groups.
problem Ensuring fair machine learning models that respect sensitive groups.
method Designing a transport-based regularizer to enforce invariance on sensitive sets.
result Certifiably fair ML models trained using SenSeI achieve improved fairness metrics.
Proposes measuring fairness through multiple stakeholder-curated stress tests.
problem Limited power of rigid fairness metrics and lack of stakeholder involvement in fairness discussions.
method Shift focus from fairness metrics to stress tests curated by stakeholders.
result Machine's performance under multiple stress tests reflects fairness.
Bayesian framework uses unlabeled data to improve fairness assessment.
problem Reliable fairness assessment with limited labeled data.
method Hierarchical latent variable model with Bayesian inference.
result Significant reduction in estimation error for fairness metrics.
The paper critiques ε-fairness, showing it can lead to unfair outcomes and proposes a utility-based approach.
problem The limitations of probabilistic fairness metrics in real-world contexts.
method Utility-based approach to measure fairness, addressing the issue of unavailable data on false negatives.
result A utility-based approach uncovers necessary actions to achieve true fairness, contrasting with traditional probability-based evaluations.
This work optimizes model performance while ensuring fairness through AUC constraints.
problem Ensuring fairness in machine learning models, especially for protected populations.
method Formulates fairness-aware machine learning model training as AUC optimization subject to fairness constraints, solves using stochastic first-order methods.
result Demonstrates effectiveness of the approach on real-world data under different fairness metrics.
This paper introduces individual fairness in clustering using f-divergence.
problem Ensuring fair clustering by treating similar individuals similarly.
method Uses f-divergence to measure statistical similarity and assigns individuals to probability distributions over cluster centers. result Provides an algorithm with provable approximation guarantee for clustering with individual fairness constraints.
Paper proposes a new FRL algorithm for continuous sensitive attributes using EIPM.
problem Existing FRL algorithms cannot handle continuous sensitive attributes.
method Introduces EIPM to assess fairness in representation space for continuous attributes and proposes FREM algorithm.
result FREM outperforms other methods in fairness evaluation for continuous sensitive attributes.
FairACE improves fairness in GNNs by balancing node performance across degree groups.
problem Degree biases in GNNs lead to unequal prediction performance among nodes with varying degrees.
method Integrates asymmetric contrastive learning with adversarial training to balance performance between high-degree and low-degree nodes.
result Significantly improves degree fairness metrics while maintaining competitive accuracy.
Develops methods to measure and reduce fairness in datasets with limited protected attribute labels.
problem Measuring and reducing fairness in datasets with limited protected attribute labels.
method Proposes methods to estimate fairness metrics and train models to limit fairness violations using probabilistic protected attribute labels.
result Our methods provide tighter bounds on true disparity and effectively reduce fairness violations with lesser fairness-accuracy trade-offs.
Multiverse analysis helps prevent fairness hacking and evaluate model design decisions.
problem Downstream effects of ADM systems depend on implicit design and evaluation decisions.
method Turn implicit decisions into explicit ones, create a grid of decision combinations, compute fairness and performance metrics.
result Decisions regarding evaluation can lead to vastly different fairness metrics for the same model.