Maximizes coding rate difference for robust, discriminative features.
problem Learning robust, discriminative features from high-dimensional data.
method Maximal Coding Rate Reduction (MCR^2) principle.
result Significantly more robust to label corruptions in classification.
Maximal initial learning rate for deep ReLU networks identified.
problem Finding the optimal initial learning rate for deep neural networks.
method Simple approach to estimate maximal initial learning rate η∗, analyzing its behavior in constant-width fully-connected ReLU networks. result Maximal initial learning rate η∗ is well predicted as a power of depth × width, with specific conditions for network width and input layer training. Study on learning rates in neural networks of varying depth.
problem Dependence of maximal update learning rate on network depth.
method Analysis of random fully connected ReLU networks with mean-field weight initialization.
result Maximal update learning rate scales like L−3/2 with network depth. New algorithm finds global maxima in multi-modal functions.
problem Finding global maxima in multi-modal functions with saddle points.
method G-PFSO algorithm using particle filter and averaging.
result G-PFSO efficiently finds global maximizer at optimal rate.
EM algorithm speeds up convergence in federated learning with heterogenous data.
problem Understanding convergence rates of federated learning algorithms under data heterogeneity.
method Characterized convergence rate of EM algorithm for FMLR model under various regimes.
result EM algorithm converges to ground truth with SNR ≥ √K in all regimes.
Semi-supervised EM improves convergence rate with labeled samples.
problem Improving convergence rate in EM algorithm with labeled and unlabeled data.
method Analysis of semi-supervised EM algorithm for Gaussian mixture models.
result Labeled samples significantly improve the convergence rate for the EM algorithm.
This paper quantifies hyperparameter transfer and finds embedding layer learning rate is key.
problem Quantifying optimal hyperparameters for large language models across scales.
method Developed three metrics to quantify hyperparameter transfer and investigated the importance of embedding layer learning rate.
result Maximal Update (μP) parameterization offers high-quality learning rate transfer compared to standard parameterization (SP).
Unified learning-rate scale for CNNs and ResNets, avoiding depth imbalance.
problem Challenges in choosing an appropriate learning rate for deep networks, especially as depth increases.
method Introduces Arithmetic-Mean μP (AM-μP), constraining network-wide average pre-activation second moment to a constant scale, combined with residual-aware He fan-in initialization. result Demonstrates a −3/2 scaling law for learning rates across depths, enabling zero-shot learning-rate transfer. The paper discusses the role of monetary policy when potential output depends on the inflation rate. If the intention of the central bank is to maximize actual output growth, then it has to be credibly committed to a strict inflation targeting rule, and to take the MOGIR (the Maximizing Output Growth Inflation Rate) as…
This work studies learning curves for revenue maximization algorithms.
problem Understanding the performance of revenue-maximizing algorithms as they learn from more data.
method Initiates the study of learning curves for revenue maximization, providing a near-complete characterization of their rate of decay.
result Learning curves for revenue maximization can decay arbitrarily slowly or almost exponentially fast, depending on the distribution and optimal revenue.
The paper tackles adaptive policy selection to maximize social welfare, achieving optimal regret bounds.
problem Maximizing social welfare through adaptive policy selection, considering both private utility and public revenue.
method The approach involves learning response functions through experimentation, deriving lower and upper bounds for regret, and using algorithms like Exp3.
result The algorithm achieves optimal regret bounds, showing that welfare maximization is harder than multi-armed bandit problems.
MaxVA improves Adam's step sizes by maximizing gradient variance.
problem Unstable or extreme adaptive learning rates in Adam.
method Maximizing the variance of gradient coordinates in Adam's running mean of squared gradients.
result Faster adaptation and more desirable convergence behaviors than Adam.
Study optimizes dividend payout strategies under fluctuating interest rates.
problem Maximizing dividends under stochastic interest rates with negative values.
method Analytical HJB approach and backward SDEs for analysis.
result Explicit optimal strategies found for both time-dependent and strategy-independent stopping times.
Proof of learning rate transfer in MLPs with μP parameterization.
problem Understanding and optimizing learning rates in neural networks with different parameterizations.
method Theoretical analysis and empirical validation of learning rate transfer in MLPs with μP, SP, and NTP parameterizations. result The optimal learning rate converges to a non-zero constant as width goes to infinity under μP, explaining learning rate transfer. Paper establishes convergence rates and concentration bounds for stochastic approximation and reinforcement learning with Markovian noise.
problem Analyzing convergence rates and concentration bounds for stochastic approximation and reinforcement learning with Markovian noise.
method Novel discretization of the mean ODE of stochastic approximation algorithms using intervals with diminishing length.
result First almost sure convergence rate and maximal concentration bound with exponential tails for contractive stochastic approximation algorithms with Markovian noise.
Stochastic AUC maximization has garnered an increasing interest due to better fit to imbalanced data classification. However, existing works are limited to stochastic AUC maximization with a linear predictive model, which restricts its predictive power when dealing with extremely complex data. In this paper, we conside…
ReduNet optimizes data compression by maximizing rate reduction in deep networks.
problem Optimizing deep networks for high-dimensional multi-class data.
method Maximizing rate reduction through iterative gradient ascent, leading to a multi-layer deep network.
result ReduNet achieves optimal linear discriminative representation and is more efficient in the spectral domain.
Scientific explanation often requires inferring maximally predictive features from a given data set. Unfortunately, the collection of minimal maximally predictive features for most stochastic processes is uncountably infinite. In such cases, one compromises and instead seeks nearly maximally predictive features. Here, …
In this paper we investigate a new class of growth rate maximization problems based on impulse control strategies such that the average number of trades per time unit does not exceed a fixed level. Moreover, we include proportional transaction costs to make the portfolio problem more realistic. We provide a Verificatio…
Recently, the decentralized optimization problem is attracting growing attention. Most existing methods are deterministic with high per-iteration cost and have a convergence rate quadratically depending on the problem condition number. Besides, the dense communication is necessary to ensure the convergence even if the …
In modern supervised learning, many deep neural networks are able to interpolate the data: the empirical loss can be driven to near zero on all samples simultaneously. In this work, we explicitly exploit this interpolation property for the design of a new optimization algorithm for deep learning, which we term Adaptive…
SGD with large learning rates can achieve better test accuracy than expected.
problem SGD with large learning rates often outperforms expected convergence bounds.
method Proved that SGD with small learning rates stays close to gradient flow path on modified loss.
result Explicitly adding an implicit regularizer to the loss improves test accuracy.
Unified framework for accelerated Perceptron and related problems.
problem Finding optimal linear threshold functions for classification.
method Modern acceleration techniques, specifically optimistic online learning.
result Improved convergence rates for various Perceptron-related problems.
In future cell-free (or cell-less) wireless networks, a large number of devices in a geographical area will be served simultaneously in non-orthogonal multiple access scenarios by a large number of distributed access points (APs), which coordinate with a centralized processing pool. For such a centralized cell-free net…
Batch Normalization (BatchNorm) is an extremely useful component of modern neural network architectures, enabling optimization using higher learning rates and achieving faster convergence. In this paper, we use mean-field theory to analytically quantify the impact of BatchNorm on the geometry of the loss landscape for …
The paper analyzes how automated market makers can retain trading fees.
problem How automated market makers can sustainably retain a portion of trading fees.
method Modeling to determine the optimal take rate for AMMs to maximize their revenue.
result AMMs can sustainably set a non-zero take rate if they have loyal trade volume.
We introduce a novel theoretical framework for Return On Investment (ROI) maximization in repeated decision-making. Our setting is motivated by the use case of companies that regularly receive proposals for technological innovations and want to quickly decide whether they are worth implementing. We design an algorithm …
We determine an explicit formula for the Laplace transform of the price of an option on a maximal interest rate when the instantaneous rate satisfies Cox-Ingersoll-Ross's model. This generalizes considerably one result of Leblanc-Scaillet.
Proposes a novel SVM model for binary classification with different misclassification costs.
problem Real-world classification problems with varying misclassification costs.
method Incorporates performance constraints in SVM formulation to seek a hyperplane with maximal margin and misclassification rates below given thresholds.
result The proposed model gives users control over misclassification rates in one class at the expense of the other.
Variational inference is becoming more and more popular for approximating intractable posterior distributions in Bayesian statistics and machine learning. Meanwhile, a few recent works have provided theoretical justification and new insights on deep neural networks for estimating smooth functions in usual settings such…
For a stochastic factor model we maximize the long-term growth rate of robust expected power utility with parameter λ∈(0,1). Using duality methods the problem is reformulated as an infinite time horizon, risk-sensitive control problem. Our results characterize the optimal growth rate, an optimal long-term trading s…
Paper improves power of conditional randomization tests.
problem Improving power of conditional randomization tests.
method Introducing a new cost function to maximize test statistic power.
result Consistently increases the number of correct discoveries.
Two insurance companies collaborate to maximize the probability of none going bankrupt.
problem Maximizing the probability of no company bankruptcy in a correlated Brownian motion model.
method Analyzing optimal strategies and deriving explicit formulas for minimal ruin probability.
result Maximizing collaboration benefits when Brownian motions are positively correlated.
Study problem-dependent rates in statistical learning theory, achieving optimal generalization error bounds.
problem Generalization error in statistical learning theory.
method Uniform localized convergence framework.
result Optimal generalization error bounds for various learning problems.
Simplifies VAE for anomaly detection using rate-distortion theory.
problem Anomaly detection in unsupervised learning systems.
method Revisit VAE from information theory, incorporate model uncertainty.
result Competitive performance on benchmark datasets.
New guarantees for ERM with adaptively collected data.
problem Failure of ERM guarantees with adaptively collected data.
method Importance sampling weighted ERM algorithm with maximal inequality.
result First generalization guarantees and fast convergence rates for adaptively collected data.
We estimate from above the rate at which a solution to the normalized Ricci flow on a closed manifold may converge to a limit soliton. Our main result implies that any solution which converges modulo diffeomorphisms to a soliton faster than any fixed exponential rate must itself be self-similar.
A new dynamic learning-rate scheme for optimization.
problem Manual tuning of learning rates in optimization is time-consuming and error-prone.
method Locally optimal stepsize estimation for dynamic learning-rate scheduling.
result Our method achieves comparable performance with minimal tuning.
In this paper we consider the problem of maximizing the Area under the ROC curve (AUC) which is a widely used performance metric in imbalanced classification and anomaly detection. Due to the pairwise nonlinearity of the objective function, classical SGD algorithms do not apply to the task of AUC maximization. We propo…
A privacy-constrained information extraction problem is considered where for a pair of correlated discrete random variables (X,Y) governed by a given joint distribution, an agent observes Y and wants to convey to a potentially public user as much information about Y as possible without compromising the amount of …
AdaScale SGD adapts learning rates for large-batch training efficiently.
problem Adapting learning rates for large-batch training to balance speed-ups and model quality.
method Adaptive learning rate adaptation based on gradient variance.
result AdaScale achieves reliable speed-ups for a wide range of batch sizes without degrading model quality.
Kernel sparsity ("dying ReLUs") and lack of diversity are commonly observed in CNN kernels, which decreases model capacity. Drawing inspiration from information theory and wireless communications, we demonstrate the intersection of coding theory and deep learning through the Grassmannian subspace packing problem in CNN…
MGD improves ML training efficiency and accuracy.
problem Optimizing large-scale ML model training configurations.
method Metagradient descent and smooth model training.
result Significantly improved model performance and learning rate schedules.
New bid shading algorithm reduces costs by 55%.
problem Minimizing costs in online advertising auctions.
method Win-rate prediction and surplus maximization using logistic regression.
result 7% more profit for advertisers compared to benchmarks.
Large learning rates work surprisingly well in standard parameterization, contrary to theory.
problem Theoretical limits of large learning rates do not match practical network behavior.
method Fine-grained analysis of learning rates and network behavior under cross-entropy loss.
result There are two distinct sub-regimes of unstable learning rates, with a controlled divergence regime where features continue to evolve.
Maximal Rate of Stepwise Uncertainty Reduction selects simulations to reduce uncertainty efficiently.
problem Efficiently estimating quantities of interest from multi-fidelity simulations.
method Bayesian sequential strategy that maximizes the ratio of expected uncertainty reduction to simulation cost.
result MR-SUR strategy unifies and provides principled approaches to develop new methods.
We investigate the problem of estimating the causal effect of a treatment on individual subjects from observational data, this is a central problem in various application domains, including healthcare, social sciences, and online advertising. Within the Neyman Rubin potential outcomes model, we use the Kullback Leibler…
Maximizing trading volume in online learning framework between traders.
problem Maximizing the total number of trades between traders with unknown valuations.
method Developed algorithms for brokers to maximize trading volume under different feedback scenarios.
result Achieved logarithmic and poly-logarithmic regret rates for different feedback models.