Unified four trade-off curves for assessing generative model proximity.
problem Quantitative assessment of proximity between two probability distributions.
method Unified four existing curves: PR, Lorenz, ROC, and Rényi divergence frontiers.
result Explicit relationship between PR and Lorenz curves with domain adaptation bounds.
We provide a general framework for characterizing the trade-off between accuracy and robustness in supervised learning. We propose a method and define quantities to characterize the trade-off between accuracy and robustness for a given architecture, and provide theoretical insight into the trade-off. Specifically we in…
Efficiently approximates fairness-accuracy trade-offs for diverse datasets.
problem Inherent trade-off between fairness and accuracy in machine learning models.
method You-Only-Train-Once (YOTO) framework for computationally efficient approximation.
result Robust methodology for auditing model fairness with statistical guarantees.
This paper compares AMMs and LOBs in exchange mechanisms, formalizing complexity vs. expressiveness trade-offs.
problem Designing efficient exchange mechanisms between assets.
method Formalizes a complexity-approximation trade-off for CFMMs and LOBs, introducing an exchange complexity measure.
result Optimally expressive mechanisms can be designed with minimal complexity, allowing for arbitrary demand curves.
Breakthroughs in machine learning are rapidly changing science and society, yet our fundamental understanding of this technology has lagged far behind. Indeed, one of the central tenets of the field, the bias-variance trade-off, appears to be at odds with the observed behavior of methods used in the modern machine lear…
Study tests whether trade-off functions are above or below benchmarks using finite samples.
problem Testing trade-off functions between unknown distributions.
method Identifies a condition for nontrivial testing, constructs a test with error guarantees, and inverts the test for confidence bands.
result Finite-sample testing is possible under specific structural assumptions about rejection regions.
The paper addresses Qini curve estimation under clustered network interference.
problem Qini curves can be biased when interference is ignored in clustered network settings.
method Proposes three estimation strategies for clustered network interference.
result Identifies the most appropriate approach based on bias-variance trade-offs.
Neural networks exhibit unimodal variance with model complexity, improving generalization.
problem The classical bias-variance trade-off does not apply to neural networks, leading to better generalization with larger models.
method Measured bias and variance of neural networks, confirmed empirically and theoretically.
result Neural networks show unimodal variance, leading to a double descent risk curve.
AIB method improves robustness against adversarial perturbations.
problem Optimizing the IB principle for better robustness and understanding compression effects.
method Proposes adversarial information bottleneck (AIB) method to optimize IB principle without explicit distribution assumptions.
result Demonstrates effectiveness in learning more invariant representations and mitigating adversarial perturbations.
Many problems that appear in biomedical decision making, such as diagnosing disease and predicting response to treatment, can be expressed as binary classification problems. The costs of false positives and false negatives vary across application domains and receiver operating characteristic (ROC) curves provide a visu…
Neural network model improves robustness of mortgage bond yield curve estimation.
problem Overfitting and instability in traditional yield curve estimation methods for small mortgage bond markets.
method Neural network framework with a new loss function for smoothness and stability.
result Empirical results show more robust and stable yield curve estimates compared to existing methods.
Random Forests don't overfit, challenging the double-descent theory.
problem The overfitting behavior of Random Forests.
method Challenged the double-descent curve theory by analyzing the performance of Random Forests and proposing a new approach to diversity in ensembles.
result Random Forests do not exhibit a double-descent curve but rather a single descent, indicating they do not overfit in the classic sense.
We consider a setup in which confidential i.i.d. samples X1,…,Xn from an unknown finite-support distribution p are passed through n copies of a discrete privatization channel (a.k.a. mechanism) producing outputs Y1,…,Yn. The channel law guarantees a local differential privacy of ε. …
New algorithm reduces FL sample and communication costs.
problem Optimizing FL for minimal samples and rounds.
method Stochastic two-sided momentum algorithm.
result Achieves near-optimal sample and communication complexities.
Theory and method for reducing prediction variance in noisy feature-subsampled ridge ensembles.
problem Reduction of prediction variance in noisy data with feature bagging.
method Developed analytical learning curves for noisy ridge ensembles, introduced heterogeneous feature ensembling.
result Subsampling shifts the double-descent peak, leading to improved performance over a single linear predictor.
Paper introduces a new power-dominance axis in estimator design.
problem Estimator design trade-off between bias and variance.
method Introduces a third power regime, `power-dominant', with an unavoidable error penalty.
result Any estimator in the `power-dominant' regime is structurally sub-optimal.
Information bottleneck (IB) is a method for extracting information from one random variable X that is relevant for predicting another random variable Y. To do so, IB identifies an intermediate "bottleneck" variable T that has low mutual information I(X;T) and high mutual information I(Y;T). The "IB curve" cha…
Single training run learns optimal VAE parameters for various β values.
problem Training VAEs with varying β values for optimal trade-off between distortion and rate.
method Introduced Multi-Rate VAE (MR-VAE) using hypernetworks to map β to optimal parameters.
result MR-VAEs can construct the full rate-distortion curve without additional training.
New method computes optimal fairness-performance trade-off without complex models.
problem Intrinsic trade-off between fairness and classifier performance.
method Computes optimal Pareto front without training complex models.
result Optimal fair representations have useful structural properties enabling efficient computation.
We consider complexity of Deep Neural Networks (DNNs) and their associated massive over-parameterization. Such over-parametrization may entail susceptibility to adversarial attacks, loss of interpretability and adverse Size, Weight and Power - Cost (SWaP-C) considerations. We ask if there are methodical ways (regulariz…
A new framework enables real-time task trade-off control.
problem Conflict between multiple related tasks in a fixed model capacity.
method Formulates MTL as a preference-conditioned multiobjective optimization problem; uses a hypernetwork-based neural network.
result A single model can handle different trade-off preferences among multiple tasks.
FlexAE addresses bias-variance trade-off in RAEs by learning latent priors.
problem Improving generation quality of deterministic AE models.
method Introducing flexibly learnable latent priors in WAEs to optimize the latent distribution.
result FlexAE achieves state-of-the-art performance in AE-based generative models.
New trade-off found between accuracy and adversarial robustness in regression.
problem Finding a balance between accuracy and robustness in regression models.
method Deriving a fundamental trade-off between standard and adversarial risk in regression with polynomial ridge functions.
result A necessary condition for achieving adversarial robustness without significant accuracy loss.
Paper explores robust regression methods and their bias-variance trade-off.
problem Understanding the trade-off between robust estimation and optimization methods.
method Examines traditional outlier-resistant robust estimation and robust optimization.
result Both methods follow converse strategies due to a bias-variance trade-off.
GeneCAI optimizes DNN compression hyper-parameters for mobile devices.
problem Efficient deployment of complex DNNs on resource-limited devices.
method GeneCAI uses genetic algorithm to learn optimal hyper-parameters.
result GeneCAI finds models with better accuracy-complexity trade-off.
This paper studies trade-offs in private prediction methods.
problem Leakage of training data information in machine learning predictions.
method Private training and private prediction methods with trade-offs.
result Private training methods outperform private prediction methods in various settings.
New insights into convergence and accuracy trade-offs in federated and meta-learning.
problem Understanding the trade-offs between convergence and accuracy in federated and meta-learning.
method Generalized local update methods, proving equivalence to first-order optimization on a surrogate loss.
result Novel convergence rates and insights into the importance of algorithmic choices in communication-limited settings.
The Lucas critique has exposed the problem of the trade-off between changes in monetary policy and structural breaks in economic time series. The search for and characterisation of such breaks has been a major econometric task ever since. We have developed an integral technique similar to CUSUM using an empirical model…
We empirically investigate the best trade-off between sparse and uniformly-weighted multiple kernel learning (MKL) using the elastic-net regularization on real and simulated datasets. We find that the best trade-off parameter depends not only on the sparsity of the true kernel-weight spectrum but also on the linear dep…
Self-training in linear models shows a U-shaped test-risk curve due to signal forgetting and denoising.
problem Understanding the dynamics of iterative self-training in high-dimensional linear regression.
method Derivation of deterministic-equivalent recursions for prediction risk and effective noise, analysis of signal forgetting and denoising effects.
result An optimal early-stopping time is determined, and a U-shaped test-risk curve is observed.
Proposes FACT, a diagnostic for understanding group fairness trade-offs.
problem Group fairness notions often conflict with each other, requiring a cost in model performance.
method Characterizes trade-offs via the fairness-confusion tensor and optimizes accuracy and fairness objectives.
result Demonstrates the use of FACT on synthetic and real datasets to understand accuracy-fairness trade-offs.
Pareto MTL finds optimal solutions for multiple tasks with different trade-offs.
problem Finding a single optimal solution for multiple conflicting tasks.
method Formulate multi-task learning as multiobjective optimization, decompose into subproblems, solve in parallel.
result Generates well-representative Pareto optimal solutions for different trade-offs.
Study shows exponential error reduction in multiclass classification without bias-variance trade-off.
problem Multiclass classification with margin conditions.
method Analysis of classification error under hard-margin conditions.
result Exponential decrease in classification error without bias-variance trade-off.
A trade-off between accuracy and fairness is almost taken as a given in the existing literature on fairness in machine learning. Yet, it is not preordained that accuracy should decrease with increased fairness. Novel to this work, we examine fair classification through the lens of mismatched hypothesis testing: trying …
Vecchia approximations provide the best accuracy-runtime trade-off for Gaussian process approximations.
problem High computational cost of Gaussian processes for large data sets.
method Systematic comparison of different Gaussian process approximations.
result Vecchia approximations consistently provide the best accuracy-runtime trade-off.
The paper explores the trade-off between bias and variance in high-dimensional models.
problem Understanding the unavoidable trade-off between bias and variance in high-dimensional statistical models.
method Proposes a general strategy to obtain lower bounds on the variance of estimators with a specified bias, and applies it to various statistical models.
result Shows the extent to which the bias-variance trade-off is unavoidable and quantifies the performance loss for methods that do not balance it.
The paper explores how noise in features can lead to benign overfitting in machine learning models.
problem Understanding the conditions for benign overfitting in machine learning models.
method Examined random feature models, specifically two-layer neural networks with fixed first layer weights, and analyzed the role of noise in features.
result Noise in features plays an important implicit regularization role in the phenomenon of benign overfitting.
Unified framework identifies nonlinear systems using characteristic curves and neural networks.
problem Balancing interpretability and flexibility in nonlinear system identification.
method Combines differential equation structure with neural networks, using characteristic curves as modular components.
result NN-CC approach outperforms other methods in complex nonlinear systems.
Despite the tremendous progress in the estimation of generative models, the development of tools for diagnosing their failures and assessing their performance has advanced at a much slower pace. Recent developments have investigated metrics that quantify which parts of the true distribution is modeled well, and, on the…
Convolutional Neural Networks (CNNs) are extremely computationally demanding, presenting a large barrier to their deployment on resource-constrained devices. Since such systems are where some of their most useful applications lie (e.g. obstacle detection for mobile robots, vision-based medical assistive technology), si…
Bias and flexibility trade off in learning algorithms.
problem Understanding the trade-off between bias and expressivity in learning algorithms.
method Measuring expressivity using entropy on algorithm outcome distributions, and deriving bounds on bias and expressivity.
result There is a necessary trade-off between bias and expressivity in learning algorithms.
Fine-tuning LLMs improves capability but harms safety, study finds.
problem Balancing capability and safety in LLM fine-tuning.
method Theoretical framework and numerical experiments for two safety-aware fine-tuning strategies.
result Characterization of fundamental limits of safety-capability trade-off in LLM fine-tuning.
Proposes Pareto efficient fairness for supervised learning models.
problem Ensuring fairness in machine learning models without sacrificing accuracy.
method Formulates a bilevel optimization problem to find Pareto efficient classifiers.
result Guaranteed solution on Pareto frontier for convex and non-convex objectives.
Reduces IB problem to a simpler, lower-dimensional problem.
problem Information bottleneck problem in high-dimensional spaces.
method Identifies sufficient statistic that factors conditional distribution, reducing IB to a lower-dimensional problem.
result Preserves full IB curve and optimal representations, making IB tractable.
Flexible deep learning models for dynamic accuracy and speed trade-offs.
problem Dynamic accuracy and speed trade-offs in real-world applications.
method Training deep neural networks with a new method allowing flexible numerical precision during inference.
result Achieved comparable accuracy to dedicated models trained at the same precision with dynamic precision settings.
This paper analyzes the trade-off between accuracy and communication in personalized federated learning.
problem The accuracy-communication trade-off in personalized federated learning.
method The paper provides a quantitative characterization of the personalization degree on the trade-off, establishing minimax optimality.
result The paper offers theoretical insights for choosing the personalization degree and validates the results on synthetic and real-world datasets.
SCORE resolves the robustness vs accuracy trade-off by redefining robust error.
problem The inherent trade-off between robustness and accuracy in adversarial training.
method SCORE defines local equivariance as the ideal robust behavior, leading to a new robust error metric.
result SCORE reconciles robustness and accuracy, improving model performance on RobustBench.
Simplifies random forests by breaking down trees into rules for better interpretability.
problem Balancing model complexity and accuracy in random forests.
method Breaking down random forest trees into individual classification rules and selecting a subset.
result A few selected rules can achieve acceptable accuracy similar to the original model, leading to simpler models.