Deeper neural networks can better approximate certain natural functions than shallower ones.
problem Approximating natural functions with neural networks.
method Depth-based separation results for feed-forward neural networks.
result Deeper networks can better approximate certain types of functions than shallower ones.
Improved neural network depth-width trade-offs via dynamical systems.
problem Expressivity of neural networks in terms of depth and width.
method Connection with dynamical systems, focusing on periodic points and Lipschitz constants.
result Sharper width lower bounds for neural networks, yielding exponential depth-width separations.
Deep forests enhance expressiveness exponentially with depth, not width or tree size.
problem Understanding the role of depth, width, and tree size in deep forest performance.
method Provided upper and lower bounds on deep forest approximation complexity.
result Depth exponentially enhances deep forest expressiveness.
New connection between DNNs and Sharkovsky's Theorem for depth-width trade-offs.
problem Understanding why some functions are hard to represent by shallow ReLU networks.
method Connection to Sharkovsky's Theorem and analysis of dynamical systems.
result Lower bounds for width needed to represent periodic functions as a function of depth.
EfficientNet improves ConvNets by scaling depth, width, and resolution effectively.
problem Improving ConvNets' accuracy and efficiency at larger scales.
method Uniformly scaling depth, width, and resolution using a compound coefficient.
result EfficientNet-B7 achieves state-of-the-art 84.3% top-1 accuracy on ImageNet.
Bayesian linear networks reveal optimal depth and width trade-offs.
problem Understanding how depth, width, and dataset size affect model quality in linear networks.
method Zero noise Bayesian inference with Gaussian weight priors and mean squared error.
result Optimal predictions at infinite depth and maximized Bayesian model evidence at infinite depth.
The bias-variance tradeoff doesn't always apply in neural networks, contradicting textbook claims.
problem The bias-variance tradeoff is not universally applicable in neural networks, contradicting textbook teachings.
method Extensive experiments and analysis on neural networks, revisiting Geman et al. (1992) experiments.
result Neural networks do not exhibit a bias-variance tradeoff when increasing network width, contradicting textbook claims.
This work generalizes bounds on the number of linear regions in CPWL NNs.
problem Determining the number of linear regions in CPWL neural networks is challenging.
method Generalized bounds on the maximal number of linear regions for arbitrary CPWL activation functions.
result Depth significantly increases the number of linear regions, but not exponentially.
Deep neural networks with piecewise-polynomial activations can approximate smooth functions and their derivatives.
problem Approximating smooth functions and their derivatives with neural networks.
method Derives the depth, width, and sparsity required for approximation in Hölder norms.
result Deep neural networks with bounded weights can approximate Hölder smooth functions and their derivatives.
A new tradeoff between regularization and sharpness improves model performance in overparameterized settings.
problem Improving model performance in overparameterized settings with minimum-norm interpolators.
method Proposes a regularization-sharpness tradeoff for overparameterized linear regression with an ℓ^p penalty.
result Empirical validation shows the tradeoff terms can distinguish performant linear interpolators.
Study the tradeoff between signal distortion and human perception over finite channels.
problem Characterize the distortion-perception tradeoff for finite channels with arbitrary metrics.
method Solve linear programming problems to compute the distortion-perception function and optimal reconstructions.
result DP function is piecewise linear in the perception index.
The paper optimizes portfolio selection with a new regret-based method.
problem Dynamic portfolio optimization with unknown tradeoff parameters.
method Regret-based selection criterion for sparse portfolios.
result Optimal sparse portfolios constructed with unknown tradeoff parameter.
The paper studies adversarial training for linear regression models.
problem Understanding the tradeoffs between robust and standard accuracy in adversarial training.
method Characterizes the fundamental tradeoff and specific adversarial training approach for linear regression with Gaussian features.
result Precise characterization of the standard and robust accuracy tradeoff in high-dimensional settings.
We perform the first study of the tradeoff space of access methods and replication to support statistical analytics using first-order methods executed in the main memory of a Non-Uniform Memory Access (NUMA) machine. Statistical analytics systems differ from conventional SQL-analytics in the amount and types of memory …
Paper explores tradeoffs in classification using tensor subspaces.
problem Supervised classification with sample, computation, and storage complexities.
method Use of tensor subspaces, particularly hierarchical Kronecker structured subspaces.
result Hierarchical Kronecker structured subspaces improve classification tradeoffs.
Paper explores tradeoff between standard and robust accuracy for latent models.
problem Tradeoff between standard accuracy and robust accuracy in adversarial training.
method Revisits adversarial training for latent models, considering Gaussian mixture and generalized linear models.
result Low-dimensional manifold structure mitigates the tradeoff between standard and robust accuracy.
Tradeoffs between risk, space, time, and data in unsupervised learning.
problem Balancing risk and computational resources in large-scale unsupervised learning.
method Developed TRAM algorithm and used coreset constructions for summarizing data.
result TRAM algorithm effectively navigates tradeoffs between risk, space, time, and data.
Proposes a new adversarial model to avoid accuracy vs. adversarial accuracy tradeoff.
problem Inherent tradeoff between accuracy and adversarial accuracy in existing adversarial robustness definitions.
method Introduces Voronoi-epsilon adversary that balances perturbation constraints.
result Voronoi-epsilon adversary avoids accuracy vs. adversarial accuracy tradeoff even with large ε. Researchers study fairness-accuracy tradeoffs in predictive models for multiple groups.
problem Understanding the tradeoff between fairness and accuracy in models serving multiple demographic groups.
method Characterizing the fairness-accuracy (FA) Pareto frontier, approximating it from limited data, and bounding the worst-case gap.
result Derivation of worst-case-optimal estimators and uniform finite-sample bounds for the entire FA frontier.
Modern neural networks show no bias-variance tradeoff with increased parameters.
problem The traditional bias-variance tradeoff does not hold in over-parameterized neural networks.
method Empirical measurements and theoretical analysis of bias and variance in modern neural networks.
result Bias and variance can decrease as the number of parameters grows in over-parameterized neural networks.
The paper explores robustness in linear regression models under adversarial attacks.
problem The impact of test-time adversarial attacks on linear regression models.
method Quantitative estimates and phase transitions analysis.
result Precise characterization of tradeoffs between adversarial robustness and accuracy.
Adversarial training can degrade standard accuracy even when optimal for robust accuracy.
problem Tradeoff between standard and robust accuracy in adversarial training.
method Analyzes adversarial training's impact on standard accuracy, even when optimal for robust accuracy.
result Even with optimal predictors, adversarial training can still degrade standard accuracy.
The paper analyzes the bias-variance tradeoff for Bregman divergences.
problem Understanding the bias-variance tradeoff for Bregman divergences.
method Analyzes the bias-variance tradeoff through operations in dual space.
result Derives several results including a generalized law of total variance and ensembling operations.
Layer-wise training for deep linear networks achieves faster convergence with optimal learning rate.
problem Training deep neural networks is challenging; layer-wise training is proposed as an alternative.
method Layer-wise training using block coordinate gradient descent (BCGD) with orthogonal-like initialization.
result The optimal learning rate guarantees the fastest decrease in loss and is applicable without prior knowledge.
Optimizes privacy-preserving data release with adversarial neural networks.
problem Minimizing distortion while concealing sensitive information in data release.
method Adversarial neural networks for randomized mechanisms and variational approximation of mutual information privacy.
result Achieves near-optimal tradeoffs between data distortion and privacy in experiments.
CDC-FM improves generative model quality-generalization tradeoff by regularizing with geometry-aware noise.
problem Tradeoff between high sample quality and memorization in deep generative models.
method Introduces Carré du champ flow matching (CDC-FM) that replaces homogeneous noise with anisotropic Gaussian noise capturing latent data manifold geometry.
result CDC-FM consistently offers better quality-generalization tradeoff across diverse datasets and architectures.
This paper explores tradeoffs between standard and adversarial risks in distributionally adversarial training.
problem Understanding the impact of adversarial training on standard risk and adversarial risk.
method Study of distributionally adversarial training with different learning settings and models.
result Derives Pareto-optimal tradeoff curves between standard and adversarial risks.
New study shows tradeoffs between compression quality, distortion, and perception.
problem Optimizing compression for low distortion often sacrifices perceptual quality.
method Adopted Blau & Michaeli's perceptual quality definition and studied the rate-distortion-perception tradeoff.
result Restricting perceptual quality to high generally requires a trade-off between rate and distortion.
New OLO algorithms use Stein's method for better performance tradeoffs.
problem Achieving optimal tradeoffs in adversarial online linear optimization.
method Operationalizing Stein's method for computationally efficient OLO algorithms.
result Additively sharp upper bounds on regret and total loss.
The paper explores the tradeoffs between fairness measures in machine learning.
problem The challenge of achieving all three fairness notions simultaneously in machine learning models.
method The approach uses partial information decomposition (PID) to analyze the relationships between fairness measures.
result Identifies the regions where fairness measures overlap and disagree, revealing potential tradeoffs.
Study shows a tradeoff between sample complexity and computational efficiency for learning halfspaces with random noise.
problem PAC learning γ-margin halfspaces with Random Classification Noise.
method Established an information-computation tradeoff and provided a simple efficient algorithm with sample complexity O(1/(γ^2 ε^2)). Also, proved lower bounds for SQ algorithms and low-degree polynomial tests.
result Inherent gap between sample complexity and computational efficiency for learning halfspaces with random noise.
Paper analyzes bias-variance tradeoff in graph Laplacian regularization.
problem Understanding the optimal regularization parameter for graph Laplacian.
method Spectral graph properties and signal-to-noise ratio parameter used to determine optimal regularization.
result Selecting mediocre regularization is often suboptimal, suggesting near-optimal performance.
This paper explores tradeoffs between invariance and sensitivity in adversarial examples.
problem Understanding the limitations of existing adversarial defenses.
method Study of invariance-based adversarial examples and their impact on model accuracy.
result Adversarial defenses against sensitivity-based attacks can harm invariance-based attacks, necessitating new approaches.
Deep neural networks with specific parameter sets can approximate smooth functions efficiently.
problem Approximating smooth functions with deep neural networks.
method Deep neural networks with ReLU activation and specific parameter sets {0,±21,±1,2} are used to approximate Cβ-smooth functions. result The constructed networks can approximate Cβ-smooth functions with parameters {0,±21,±1,2} efficiently, achieving the same convergence rate as sparse networks with parameters in [−1,1]. Pruned neural networks' error scales predictably with architecture and task.
problem Understanding the predictability of pruning across different scales and architectures.
method Functionally approximated the error of pruned networks, showing it is predictable in terms of invariant tying width, depth, and pruning level.
result The error of pruned networks follows a scaling law with interpretable coefficients that depend on architecture and task.
This paper merges deterministic policy gradient estimations to improve deep reinforcement learning performance.
problem The bias-variance tradeoff in estimating and using policy gradients for deep reinforcement learning.
method Introduces elite policy gradients and a two-step merging method to balance bias-variance tradeoffs.
result Two-step merging outperforms interpolation merging and state-of-the-art algorithms on benchmark control tasks.
The paper analyzes the tradeoffs between accuracy and invariance in learning representations.
problem Achieving both accuracy and invariance in machine learning models.
method Information theoretic analysis of classification and regression settings.
result Characterization of the accuracy and invariance achievable by any representation of the data.
Paper improves privacy-accuracy balance in federated learning.
problem Privacy-accuracy tradeoffs in federated learning.
method Personalized federated learning with joint differential privacy.
result Coordination of local and centralized learning improves accuracy while maintaining privacy.
Deep neural networks with various activation functions can approximate Hölder smooth functions.
problem Expressivity of deep neural networks with general activation functions.
method Investigates approximation ability of deep neural networks with a broad class of activation functions, including Hölder smooth functions.
result Derives the required depth, width, and sparsity of deep neural networks to approximate Hölder smooth functions.
Improves early stopping in deep networks by adjusting stepsizes.
problem Epoch-wise double descent in deep networks.
method Analytical and empirical study of bias-variance tradeoffs in different network layers.
result Eliminating epoch-wise double descent through adjusting stepsizes of different layers improves early stopping performance.
WGANs improve probability distribution approximation with depth and width trade-offs.
problem Approximating complex probability distributions accurately.
method Wasserstein GANs with GroupSort discriminators, quantified generalization bound.
result High-capacity discriminators are crucial for WGANs' performance.
Algorithm balances online and offline data for linear bandits.
problem Online learning with an offline dataset in linear bandits.
method Proposes a linear bandit algorithm that uses offline data early and increasingly favors exploration as the horizon grows.
result Establishes regret bounds showing competitive performance with both purely online and offline solutions.
This work explores feature learning tradeoffs in neural networks.
problem Resource tradeoffs in neural feature learning.
method Theoretical and experimental investigation of offline sparse parity learning.
result Width improves sample efficiency in sparse feature learning.
The stochastic block model guides community detection limits and algorithms.
problem Understanding community detection limits in stochastic block models.
method Information-theoretic and computational tradeoffs, algorithms derivation.
result Phase transitions and mutual information tradeoffs for community detection.
The paper bounds the excess risk of deep neural networks for weakly dependent processes.
problem Learning with weakly dependent data using deep neural networks.
method Approximation of smooth functions by deep neural networks and a bound on excess risk.
result The excess risk bound for deep learning under weak dependence is close to O(n−1/2) for sufficiently smooth functions. New method selects neural network architectures without needing data.
problem Choosing efficient deep neural network architectures.
method Developed the deep frame potential to quantify network capacity.
result Deep frame potential correlates with generalization error.
New insights into bias-variance tradeoff for data-driven optimization under local misspecification.
problem Understanding the relative performance of SAA, IEO, and ETO under local misspecification.
method Developed a local misspecification perspective using contiguity theory in statistics.
result Explicit expressions for decision bias and geometric understanding of variance.
New method quantifies redundant information using information bottleneck.
problem Quantifying redundant information among multiple sources.
method Formulated as an information bottleneck problem, termed redundancy bottleneck.
result Extracts information that best predicts the target without revealing source identity.