Study capacity constraints in continual learning with a simple model.
problem Understanding optimal resource allocation for agents with limited memory and compute resources.
method Analyzes a capacity-constrained linear-quadratic-Gaussian (LQG) sequential prediction problem and demonstrates optimal capacity allocation strategies.
result Derives a solution to the capacity-constrained LQG sequential prediction problem and shows how to optimally allocate capacity across sub-problems in the steady state.
Learning capacity measures model complexity, correlating with test loss and sample size.
problem Understanding model complexity and its relation to test performance.
method Formal correspondence between thermodynamics and inference; learning capacity as a measure of effective dimensionality.
result Learning capacity correlates with test loss and is a small fraction of model parameters.
Improves online learning algorithms for functional models with capacity assumptions.
problem Convergence rates of online stochastic gradient descent algorithms for functional linear models.
method Characterizations of slope function regularity, kernel space capacity, and sampling process covariance operator.
result Capacity assumptions can alleviate saturation of convergence rates as function regularity increases.
Study online learning with delays and capacity constraints, achieving optimal regret bounds.
problem Online learning with delays and capacity constraints.
method Novel scheduling and preemptive techniques, matching upper and lower bounds.
result Achieves optimal regret bounds across all capacity levels.
Study proposes local effective dimension to measure model capacity and generalization error.
problem Capturing the generalization power of machine learning models.
method Proposes local effective dimension as a capacity measure.
result Local effective dimension bounds the generalization error and correlates well with it.
Memory capacity of DAM scales exponentially with feature separation, unaffected by correlations.
problem Understanding how feature correlations impact DAM's capacity.
method Developed an empirical framework to analyze DAM's capacity under varying feature correlations and pattern separations.
result Memory capacity scales exponentially with feature separation, unaffected by correlations.
New algorithm for shareable arms with load-dependent rewards in stochastic bandits.
problem Learning optimal play strategy with shareable finite-capacity arms in stochastic bandits.
method Developed a capacity estimator and online learning algorithm for MP-MAB with shareable arms.
result Regret upper bound matches the lower bound, validating the algorithm's performance.
Dropout controls model capacity in deep learning and matrix completion.
problem Controlling model capacity in deep learning and matrix completion problems.
method Investigates dropout's effect on model capacity and Rademacher complexity.
result Dropout induces a regularizer that controls model capacity in expectation.
Recurrent neural networks are powerful models for processing sequential data, but they are generally plagued by vanishing and exploding gradient problems. Unitary recurrent neural networks (uRNNs), which use unitary recurrence matrices, have recently been proposed as a means to avoid these issues. However, in previous …
Model predicts COVID-19 growth in Senegal, highlighting health care capacity importance.
problem Impact of health care capacity on COVID-19 growth in Senegal.
method Compartmental model with logistic growth health care capacity, machine learning projection.
result Condition to avoid overwhelming health care system provided.
Paper tackles inventory management with deep learning, improving performance and adherence to constraints.
problem Managing inventory with limited resources and constraints.
method Proposes a novel method to sample from a distribution of possible constraint paths, extends exo-IDP formulation, introduces neural coordinator, and uses modified DirectBackprop algorithm.
result Deep reinforcement learning policies with a neural coordinator outperform classic baselines in terms of performance and adherence to constraints.
Study shows how correlations between neural activity affect classification capacity.
problem Understanding how correlations between neural activity impact classification performance.
method Calculated the capacity of neural activity on spherical manifolds with and without correlations between centroids and axes.
result Introducing correlations between neural activity centroids pushes spheres closer together, while correlations between axes shrink their radii, revealing a duality between correlations and geometry in classification.
Introduces Rashomon Capacity to measure predictive multiplicity in probabilistic classifiers.
problem Predictive multiplicity in classification models leading to unjustified decisions.
method Introduces Rashomon Capacity, a metric for probabilistic classifiers, and provides a rigorous derivation.
result Rashomon Capacity captures nuanced score variations and provides strategies for disclosing conflicting models.
A latent function decomposition method is proposed for forecasting the capacity of lithium-ion battery cells. The method uses the Multi-Output Gaussian Process, a generative machine learning framework for multi-task and transfer learning. The MCGP decomposes the available capacity trends from multiple battery cells int…
New study shows how model complexity affects test risk, challenging classical theory.
problem Understanding how test risk scales with model complexity for large over-parametrized deep networks.
method Developed norm-based capacity measures for random features based estimators, providing precise characterization of estimator's norm concentration and test error.
result Predicted learning curve shows a phase transition from under- to over-parameterization, confirming classical U-shaped behavior with appropriate capacity measures.
Study on neural networks' storage capacity and solution space structure.
problem Understanding the storage capacity and solution space structure of neural networks.
method Replica method from statistical physics.
result Storage capacity per parameter remains finite even with infinite width and weights exhibit negative correlations.
There are (at least) three approaches to quantifying information. The first, algorithmic information or Kolmogorov complexity, takes events as strings and, given a universal Turing machine, quantifies the information content of a string as the length of the shortest program producing it. The second, Shannon information…
New analysis tightens memory capacity of Hopfield models using spherical codes.
problem Optimizing memory capacity in modern Hopfield models and Kernelized Hopfield Models.
method Connecting Hopfield models to spherical codes in information theory, establishing an optimal capacity bound and a sub-linear algorithm.
result First tight and optimal asymptotic memory capacity for modern Hopfield models, matching known lower bounds.
Study on Privileged ERM showing limitations and providing capacity analysis.
problem Improving classification accuracy with privileged information.
method Theoretical analysis of Privileged ERM using VC dimension and generalization bounds.
result Worst-case guarantees for Privileged ERM cannot improve over standard ERM unless privileged information capacity is similar or smaller.
New complexity measure helps in agnostic reinforcement learning with or without access to MDP dynamics.
problem Understanding the number of rounds needed to learn an ε-suboptimal policy in unknown MDPs.
method Introducing spanning capacity as a new complexity measure and developing POPLER algorithm.
result There is a separation between generative and online access models for agnostic learnability.
RAF model explains neural networks' dual rule learning and fact memorization.
problem Understanding how neural networks learn rules and memorize facts simultaneously.
method Introduces the Rules-and-Facts (RAF) model to bridge generalization and memorization.
result Characterizes conditions for simultaneous rule learning and fact memorization in neural networks.
Normalization layers control deep neural network capacity, improving stability and generalization.
problem Excessive capacity in deep neural networks leads to overfitting and poor generalization.
method Developed a theoretical framework to explain normalization's role in capacity control.
result Normalization layers reduce the Lipschitz constant exponentially, smoothing the loss landscape and enhancing generalization.
Framework for understanding overfitting and underfitting using information theory.
problem Understanding and preventing overfitting and underfitting in machine learning.
method Information-theoretic framework measuring algorithm capacity and dataset information transfer.
result Upper-bounding algorithm capacity and establishing its relationship to machine learning quantities.
Study characterizes memory capacity of quantum reservoirs using transmon qubits.
problem Understanding the memory capacity of quantum reservoirs built with transmon qubits.
method Characterized memory capacity of quantum reservoirs using transmon qubits from IBM, focusing on NMSE and topology complexity.
result Found a peak in memory capacity for configurations with n-1 self-loops, suggesting optimal design for forecasting tasks.
We experimentally achieve a 19% capacity gain per Watt of electrical supply power in a 12-span link by eliminating gain flattening filters and optimizing launch powers using machine learning by deep neural networks in a massively parallel fiber context.
A variety of large-scale machine learning problems can be cast as instances of constrained submodular maximization. Existing approaches for distributed submodular maximization have a critical drawback: The capacity - number of instances that can fit in memory - must grow with the data set size. In practice, while one c…
Remove symmetries to improve model optimization and performance.
problem Symmetries in loss functions trap models in low-capacity states, hindering training and optimization.
method Proposes syre, a simple algorithm to remove symmetries in neural networks.
result Removing symmetries correlates well with improved optimization and performance.
High-capacity neural network ensembles often benefit more from high-capacity models than from increased diversity.
problem The performance of high-capacity neural network ensembles is often harmed by interventions that promote predictive diversity.
method A large-scale study of nearly 600 neural network classification ensembles, examining various interventions and architectures.
result Discouraging predictive diversity can be benign in large-network ensembles, and higher-capacity models often yield better performance than diverse architectures.
We study various capacities on compact Kähler manifolds which generalize the Bedford-Taylor Monge-Ampère capacity. We then use these capacities to study the existence and the regularity of solutions of complex Monge-Ampère equations.
New measure FTC quantifies how much a ReLU network can fine-tune.
problem Analyzing memorization capacity in fine-tuned neural networks.
method Defined Fine-Tuning Capacity (FTC) for additive fine-tuning of ReLU networks.
result Upper and lower bounds on FTC for 2 and 3-layer ReLU networks.
Study derives error decay rates for kernel classification under source and capacity conditions.
problem Understanding prediction error decay rates for real data sets.
method Derived decay rates for misclassification error under Gaussian design for SVM and ridge classification.
result Rates accurately describe learning curves for data sets satisfying source and capacity conditions.
Solves a discrete logarithmic Minkowski problem for electrostatic p-capacity.
problem Characterize measures generated by electrostatic p-capacity.
method Solves the discrete logarithmic Minkowski problem for 1 < p < n.
result Solves the discrete logarithmic Minkowski problem for measures in general position.
A long standing open problem in the theory of neural networks is the development of quantitative methods to estimate and compare the capabilities of different architectures. Here we define the capacity of an architecture by the binary logarithm of the number of functions it can compute, as the synaptic weights are vari…
The paper tackles imbalanced classification under operational constraints, proposing a framework to maximize sensitivity.
problem Detecting minority class observations under severe class imbalance and operational constraints.
method Formal classification framework under capacity constraints, maximizing sensitivity while respecting a user-defined label limit.
result The optimal classifier under capacity constraints is equivalent to the Bayes classifier with reweighted prior probabilities.
CapOptix uses options theory to price capacity in electricity markets.
problem Traditional capacity market designs fail to account for risk and price shocks.
method Interprets capacity commitments as reliability options and uses Markov Regime Switching Process.
result CapOptix provides more accurate pricing of capacity premia compared to existing mechanisms.
An online learning framework optimizes pricing and capacity in service systems.
problem Optimizing pricing and capacity in dynamic service systems.
method Gradient-based Online Learning in Queue (GOLiQ) framework.
result GOLiQ achieves logarithmic regret bound and improves service provider's performance.
In this article, we propose the notion of the general p-affine capacity and prove some basic properties for the general p-affine capacity, such as affine invariance and monotonicity. The newly proposed general p-affine capacity is compared with several classical geometric quantities, e.g., the volume, the p-var…
The paper extends IPC framework to stationary physical systems and validates it with a photonic system.
problem Characterizing the computational capabilities of stationary physical systems in a principled, data-efficient way.
method Extended IPC framework, established fundamental results, derived asymptotic bias, introduced data-efficient estimation methods.
result IPC strongly correlates with machine-learning performance and provides a reliable estimate of system dimensionality.
While symplectic manifolds have no local invariants, they do admit many global numerical invariants. Prominent among them are the so-called symplectic capacities. Different capacities are defined in different ways, and so relations between capacities often lead to surprising relations between different aspects of sympl…
Study excess capacity in neural networks using Rademacher complexity.
problem Understanding how much capacity deep networks have beyond what's needed for classification.
method Unified Rademacher complexity bounds for function composition and convolutional layers, considering Lipschitz constants and initialization norms.
result There is substantial excess capacity per task, and capacity can be kept similar across different tasks.
Study rigidity by logarithmic capacity and related functions.
problem Rigidity phenomena in kernel functions and capacities.
method Exploration of Bergman kernel, logarithmic capacity, Green's function, and Euclidean distance/volume.
result Established rigidity theorems by logarithmic capacity.
Study binary perceptrons' capacity using random duality theory.
problem Characterize the capacity of binary perceptrons with general thresholds.
method Utilized fully lifted random duality theory (fl RDT) to characterize the capacity.
result Characterizations match replica symmetry breaking predictions and uncover the capacity for zero-threshold scenario.
New complete panel dataset for LMICs helps analyze innovation and development.
problem Lack of complete data for empirical analyses in LMICs.
method Predictive Mean Matching multiple imputation technique.
result Created a large dataset of 47 variables for 82 LMICs from 2005-2019.
Upper bounds for Lagrangian capacities of Liouville domains
problem Lagrangian capacity of Liouville domains
method Using S1-equivariant techniques result Extremal Lagrangian torus on the boundary of ellipsoid
Proves local maximizers for higher Ekeland-Hofer capacities in 4D star-shaped domains.
problem Finding local maximizers for higher Ekeland-Hofer capacities in specific domains.
method Analogous to 4D local Viterbo conjecture, proving maximizers for rational ellipsoids.
result Local maximizers of the k-th Ekeland-Hofer capacities are symplectomorphic to rational ellipsoids.
The Variational AutoEncoder (VAE) learns simultaneously an inference and a generative model, but only one of these models can be learned at optimum, this behaviour is associated to the ELBO learning objective, that is optimised by a non-informative generator. In order to solve such an issue, we provide a learning objec…
Capacity-Constrained Online Convex Optimization with Delayed Feedback
problem Online learning with delayed feedback under a hard capacity constraint
method Reduction to a delayed and weighted OCO problem using a scheduler
result First regret guarantees for capacity-constrained OCO under convex and strongly convex losses
Develops a theory for mth order p-affine capacity for convex bodies containing the origin.
problem Defines and studies the mth order p-affine capacity for convex bodies containing the origin.
method Provides equivalent definitions, proves properties, and establishes inequalities.
result Establishes inequalities comparing to other geometric measures.