The bias-variance tradeoff doesn't always apply in neural networks, contradicting textbook claims.
problem The bias-variance tradeoff is not universally applicable in neural networks, contradicting textbook teachings.
method Extensive experiments and analysis on neural networks, revisiting Geman et al. (1992) experiments.
result Neural networks do not exhibit a bias-variance tradeoff when increasing network width, contradicting textbook claims.
Paper analyzes bias-variance tradeoff in graph Laplacian regularization.
problem Understanding the optimal regularization parameter for graph Laplacian.
method Spectral graph properties and signal-to-noise ratio parameter used to determine optimal regularization.
result Selecting mediocre regularization is often suboptimal, suggesting near-optimal performance.
Paper solves NGCA for discrete distributions using LLL method.
problem Learning hidden non-Gaussian components in discrete distributions.
method Utilizes LLL lattice basis reduction method.
result Sample and computationally efficient algorithm for NGCA in discrete distributions.
The paper analyzes the tradeoff between bias and overfitting in reinforcement learning with partial observability.
problem Analyzing the tradeoff between asymptotic bias and overfitting in reinforcement learning with partial observability.
method Theoretical analysis and empirical illustration using truncated history of observations and function approximators.
result A smaller state representation decreases the risk of overfitting, but potentially increases asymptotic bias.
Modern neural networks show no bias-variance tradeoff with increased parameters.
problem The traditional bias-variance tradeoff does not hold in over-parameterized neural networks.
method Empirical measurements and theoretical analysis of bias and variance in modern neural networks.
result Bias and variance can decrease as the number of parameters grows in over-parameterized neural networks.
The paper explores the tradeoffs between fairness measures in machine learning.
problem The challenge of achieving all three fairness notions simultaneously in machine learning models.
method The approach uses partial information decomposition (PID) to analyze the relationships between fairness measures.
result Identifies the regions where fairness measures overlap and disagree, revealing potential tradeoffs.
The paper analyzes the tradeoffs between accuracy and invariance in learning representations.
problem Achieving both accuracy and invariance in machine learning models.
method Information theoretic analysis of classification and regression settings.
result Characterization of the accuracy and invariance achievable by any representation of the data.
Paper characterizes fairness vs. accuracy tradeoff in classification.
problem Mitigating bias in machine learning models.
method Characterizes the tradeoff between fairness and accuracy, provides a post-processing algorithm.
result Post-processing algorithm yields optimal fair classifier when score is Bayes optimal.
New insights into bias-variance tradeoff for data-driven optimization under local misspecification.
problem Understanding the relative performance of SAA, IEO, and ETO under local misspecification.
method Developed a local misspecification perspective using contiguity theory in statistics.
result Explicit expressions for decision bias and geometric understanding of variance.
The paper clarifies fairness vs. accuracy in criminal justice risk assessments.
problem Lack of conceptual precision in discussions of fairness in criminal justice risk assessments.
method Integrated examination of fairness and accuracy using criminology, computer science, and statistics.
result There are at least six kinds of fairness, some incompatible with accuracy.
The paper analyzes Adam and SGD in nonstationary optimization, revealing tradeoffs between noise and drift.
problem Analyzing Adam and SGD in nonstationary optimization problems.
method Theoretical analysis of Adam and SGD under non-stationary stochastic objectives, separating two regimes.
result Characterizes the tradeoff between noise and drift in Adam and SGD, revealing when adaptive step-sizing is beneficial or harmful.
Hyperbolic embeddings reduce dimensions for hierarchical data with high precision.
problem Embedding hierarchical data structures like synonym or type hierarchies efficiently.
method Combinatorial construction and hyperbolic multidimensional scaling (h-MDS) for metric spaces.
result Hyperbolic embeddings achieve high precision with few dimensions, e.g., 0.989 MAP with only 2 dimensions on WordNet.
New method trains GFlowNets from partial episodes to improve convergence and stability.
problem Improving convergence and stability of GFlowNets training.
method Introducing SubTB(λ) for GFlowNet training from partial action subsequences. result SubTB(λ) accelerates GFlowNet convergence and enables training in longer action sequences. A new approach selects tuning parameters for embedding methods.
problem Difficulty in selecting tuning parameters for embedding methods.
method Minimize a stress notion to supervise tuning parameter selection.
result Uncover a new bias--variance tradeoff phenomenon.
Paper analyzes the tradeoff between a discriminator's ability to identify true distribution and its ability to generalize.
problem Discrimination-generalization tradeoff in GANs.
method Analyzes the linear span of discriminator sets and their relationship to generalization and discriminative ability.
result Discriminator sets with dense linear span are both discriminative and generalizable.
A new tradeoff between regularization and sharpness improves model performance in overparameterized settings.
problem Improving model performance in overparameterized settings with minimum-norm interpolators.
method Proposes a regularization-sharpness tradeoff for overparameterized linear regression with an ℓ^p penalty.
result Empirical validation shows the tradeoff terms can distinguish performant linear interpolators.
Study the tradeoff between signal distortion and human perception over finite channels.
problem Characterize the distortion-perception tradeoff for finite channels with arbitrary metrics.
method Solve linear programming problems to compute the distortion-perception function and optimal reconstructions.
result DP function is piecewise linear in the perception index.
Paper proposes SRA algorithm for online learning robustness and adaptivity.
problem Quantifying and evaluating tradeoff between robustness and adaptivity in online learning.
method SRA algorithm using biased stochastic approximation scheme with adaptive threshold.
result SRA algorithm provides superior performance in synthetic and real datasets.
The paper optimizes portfolio selection with a new regret-based method.
problem Dynamic portfolio optimization with unknown tradeoff parameters.
method Regret-based selection criterion for sparse portfolios.
result Optimal sparse portfolios constructed with unknown tradeoff parameter.
The paper studies adversarial training for linear regression models.
problem Understanding the tradeoffs between robust and standard accuracy in adversarial training.
method Characterizes the fundamental tradeoff and specific adversarial training approach for linear regression with Gaussian features.
result Precise characterization of the standard and robust accuracy tradeoff in high-dimensional settings.
We perform the first study of the tradeoff space of access methods and replication to support statistical analytics using first-order methods executed in the main memory of a Non-Uniform Memory Access (NUMA) machine. Statistical analytics systems differ from conventional SQL-analytics in the amount and types of memory …
New L1 regularization controls neural network generalization error and sparsifies input dimensions.
problem Selecting the optimal number of hidden neurons in neural networks.
method Theoretical analysis of L1 regularization in two-layer neural networks. result Appropriate L1 regularization leads to near minimax optimal generalization risk bounds. Graphical models have gained a lot of attention recently as a tool for learning and representing dependencies among variables in multivariate data. Often, domain scientists are looking specifically for differences among the dependency networks of different conditions or populations (e.g. differences between regulatory …
Regression models can interpolate noisy data and still perform well, contrary to the bias-variance tradeoff.
problem Understanding why overparametrized models can generalize well despite the bias-variance tradeoff.
method Analysis of minimum norm solutions and ridge regression, focusing on the smallest singular value of the regression matrix.
result Testing error exhibits double descent behavior as model order increases, contrary to the classical bias-variance tradeoff.
Paper explores tradeoffs in classification using tensor subspaces.
problem Supervised classification with sample, computation, and storage complexities.
method Use of tensor subspaces, particularly hierarchical Kronecker structured subspaces.
result Hierarchical Kronecker structured subspaces improve classification tradeoffs.
Paper explores tradeoff between standard and robust accuracy for latent models.
problem Tradeoff between standard accuracy and robust accuracy in adversarial training.
method Revisits adversarial training for latent models, considering Gaussian mixture and generalized linear models.
result Low-dimensional manifold structure mitigates the tradeoff between standard and robust accuracy.
Tradeoffs between risk, space, time, and data in unsupervised learning.
problem Balancing risk and computational resources in large-scale unsupervised learning.
method Developed TRAM algorithm and used coreset constructions for summarizing data.
result TRAM algorithm effectively navigates tradeoffs between risk, space, time, and data.
Proposes a new adversarial model to avoid accuracy vs. adversarial accuracy tradeoff.
problem Inherent tradeoff between accuracy and adversarial accuracy in existing adversarial robustness definitions.
method Introduces Voronoi-epsilon adversary that balances perturbation constraints.
result Voronoi-epsilon adversary avoids accuracy vs. adversarial accuracy tradeoff even with large ε. Researchers study fairness-accuracy tradeoffs in predictive models for multiple groups.
problem Understanding the tradeoff between fairness and accuracy in models serving multiple demographic groups.
method Characterizing the fairness-accuracy (FA) Pareto frontier, approximating it from limited data, and bounding the worst-case gap.
result Derivation of worst-case-optimal estimators and uniform finite-sample bounds for the entire FA frontier.
Sum-of-Squares lower bound shows NGCA requires more samples than known algorithms.
problem Finding a non-Gaussian direction in a high-dimensional dataset.
method Sum-of-Squares (SoS) framework to prove lower bounds.
result First super-constant degree SoS lower bound for NGCA.
The paper explores robustness in linear regression models under adversarial attacks.
problem The impact of test-time adversarial attacks on linear regression models.
method Quantitative estimates and phase transitions analysis.
result Precise characterization of tradeoffs between adversarial robustness and accuracy.
Adversarial training can degrade standard accuracy even when optimal for robust accuracy.
problem Tradeoff between standard and robust accuracy in adversarial training.
method Analyzes adversarial training's impact on standard accuracy, even when optimal for robust accuracy.
result Even with optimal predictors, adversarial training can still degrade standard accuracy.
The paper analyzes the bias-variance tradeoff for Bregman divergences.
problem Understanding the bias-variance tradeoff for Bregman divergences.
method Analyzes the bias-variance tradeoff through operations in dual space.
result Derives several results including a generalized law of total variance and ensembling operations.
Optimizes privacy-preserving data release with adversarial neural networks.
problem Minimizing distortion while concealing sensitive information in data release.
method Adversarial neural networks for randomized mechanisms and variational approximation of mutual information privacy.
result Achieves near-optimal tradeoffs between data distortion and privacy in experiments.
CDC-FM improves generative model quality-generalization tradeoff by regularizing with geometry-aware noise.
problem Tradeoff between high sample quality and memorization in deep generative models.
method Introduces Carré du champ flow matching (CDC-FM) that replaces homogeneous noise with anisotropic Gaussian noise capturing latent data manifold geometry.
result CDC-FM consistently offers better quality-generalization tradeoff across diverse datasets and architectures.
This paper explores tradeoffs between standard and adversarial risks in distributionally adversarial training.
problem Understanding the impact of adversarial training on standard risk and adversarial risk.
method Study of distributionally adversarial training with different learning settings and models.
result Derives Pareto-optimal tradeoff curves between standard and adversarial risks.
New study shows tradeoffs between compression quality, distortion, and perception.
problem Optimizing compression for low distortion often sacrifices perceptual quality.
method Adopted Blau & Michaeli's perceptual quality definition and studied the rate-distortion-perception tradeoff.
result Restricting perceptual quality to high generally requires a trade-off between rate and distortion.
New OLO algorithms use Stein's method for better performance tradeoffs.
problem Achieving optimal tradeoffs in adversarial online linear optimization.
method Operationalizing Stein's method for computationally efficient OLO algorithms.
result Additively sharp upper bounds on regret and total loss.
Study shows a tradeoff between sample complexity and computational efficiency for learning halfspaces with random noise.
problem PAC learning γ-margin halfspaces with Random Classification Noise.
method Established an information-computation tradeoff and provided a simple efficient algorithm with sample complexity O(1/(γ^2 ε^2)). Also, proved lower bounds for SQ algorithms and low-degree polynomial tests.
result Inherent gap between sample complexity and computational efficiency for learning halfspaces with random noise.
This paper provides performance guarantees for neural estimation of statistical distances.
problem Developing performance guarantees for neural estimation of statistical distances.
method Non-asymptotic error bounds using function approximation theorems and empirical process theory.
result Established a fundamental tradeoff between approximation and estimation errors in neural estimation of statistical distances.
The paper explores robust classifiers for imbalanced Gaussian data.
problem Adversarial robustness in machine learning with imbalanced data.
method Developed exact and approximate Bayes-optimal robust classifiers for Gaussian classification problems.
result Revealed fundamental tradeoffs between standard and robust accuracy.
This paper explores tradeoffs between invariance and sensitivity in adversarial examples.
problem Understanding the limitations of existing adversarial defenses.
method Study of invariance-based adversarial examples and their impact on model accuracy.
result Adversarial defenses against sensitivity-based attacks can harm invariance-based attacks, necessitating new approaches.
New method estimates shape distance in neural representations with limited data.
problem Measuring geometric similarity between high-dimensional network representations.
method Method-of-moments estimator with tunable bias-variance tradeoff.
result New estimator achieves lower bias than standard methods in high-dimensional settings.
This paper merges deterministic policy gradient estimations to improve deep reinforcement learning performance.
problem The bias-variance tradeoff in estimating and using policy gradients for deep reinforcement learning.
method Introduces elite policy gradients and a two-step merging method to balance bias-variance tradeoffs.
result Two-step merging outperforms interpolation merging and state-of-the-art algorithms on benchmark control tasks.
Unified approach for fair classification with overlapping groups.
problem Ensuring fairness across multiple overlapping groups in prediction problems.
method Probabilistic population analysis leading to Bayes-optimal classifier, unifying existing methods.
result Outperforms baselines in fairness-performance tradeoff on real datasets.
IDT learns human preferences from uncertain decisions, even when humans are suboptimal.
problem Learning human preferences from uncertain and suboptimal decisions.
method Inverse decision theory (IDT) framework, statistical analysis of IDT, characterizing sample complexity.
result Learning preferences is easier when decisions are more uncertain, even if humans are suboptimal.
Paper improves privacy-accuracy balance in federated learning.
problem Privacy-accuracy tradeoffs in federated learning.
method Personalized federated learning with joint differential privacy.
result Coordination of local and centralized learning improves accuracy while maintaining privacy.
A fast method for sparse PCA reduces computation time.
problem Time-consuming implementation of SPCA on high-dimensional data.
method Subspace projections using Household QR factorization for efficient deflation.
result Developed SPCA-SP method maintains good tradeoffs between various criteria.