Modeling alignment as resource-limited cognitive processes, researchers derive performance bounds.
problem Systematic deviations in feedback-based alignment of large language models.
method Modeling alignment as a two-stage cascade UoHoY given S, with cognitive and total capacities. result Capacity-coupled Alignment Performance Interval derived from Fano and PAC-Bayes bounds.
Model for cross-border markets with limited transmission capacities.
problem Limited transmission capacities between two countries' markets.
method Developed a regime-switching process model with high-frequency approximation.
result Analytic tractability allows computation of key market quantities.
Capacity analysis has been recently introduced as a way to analyze how linear models distribute their modelling capacity across the input space. In this paper, we extend the notion of capacity allocation to the case of neural networks with non-linear layers. We show that under some hypotheses the problem is equivalent …
Study shows how activation functions impact the storage capacity of treelike neural networks.
problem Understanding the role of activation functions in neural network expressive power.
method Analysis of treelike two-layer networks with various activation functions in the infinite-width limit.
result Activation functions affect storage capacity and robustness, with nonlinearity increasing capacity and decreasing robustness.
Study on Privileged ERM showing limitations and providing capacity analysis.
problem Improving classification accuracy with privileged information.
method Theoretical analysis of Privileged ERM using VC dimension and generalization bounds.
result Worst-case guarantees for Privileged ERM cannot improve over standard ERM unless privileged information capacity is similar or smaller.
Study capacity constraints in continual learning with a simple model.
problem Understanding optimal resource allocation for agents with limited memory and compute resources.
method Analyzes a capacity-constrained linear-quadratic-Gaussian (LQG) sequential prediction problem and demonstrates optimal capacity allocation strategies.
result Derives a solution to the capacity-constrained LQG sequential prediction problem and shows how to optimally allocate capacity across sub-problems in the steady state.
This research proves guarantees on sequence models' generalization to longer and novel sequences.
problem Generalization to longer sequences and novel token combinations in sequence models.
method Provable guarantees on length and compositional generalization for various sequence models.
result Limited capacity models achieve both length and compositional generalization with diverse training distributions.
Through using the semidiameter (in connection to: the mean radius and surface radius) of a convex closed hypersurface in Rn≥2 as an sharp upper bound of the variational (1,n)∋p-capacity radius, this paper settles a restriction/variant of S.-T. Yau's \cite[Problem 59]{Yau} from the surface area to t…
Memory capacity of DAM scales exponentially with feature separation, unaffected by correlations.
problem Understanding how feature correlations impact DAM's capacity.
method Developed an empirical framework to analyze DAM's capacity under varying feature correlations and pattern separations.
result Memory capacity scales exponentially with feature separation, unaffected by correlations.
Data selection boosts fact memorization in language models.
problem Language models struggle to accurately memorize factual knowledge.
method Formalizes fact memorization, proposes data selection schemes based on training loss.
result Data selection boosts fact accuracy to model capacity and improves performance.
Paper tackles inventory management with deep learning, improving performance and adherence to constraints.
problem Managing inventory with limited resources and constraints.
method Proposes a novel method to sample from a distribution of possible constraint paths, extends exo-IDP formulation, introduces neural coordinator, and uses modified DirectBackprop algorithm.
result Deep reinforcement learning policies with a neural coordinator outperform classic baselines in terms of performance and adherence to constraints.
We study the computational capacity of a model neuron, the Tempotron, which classifies sequences of spikes by linear-threshold operations. We use statistical mechanics and extreme value theory to derive the capacity of the system in random classification tasks. In contrast to its static analog, the Perceptron, the Temp…
A variety of large-scale machine learning problems can be cast as instances of constrained submodular maximization. Existing approaches for distributed submodular maximization have a critical drawback: The capacity - number of instances that can fit in memory - must grow with the data set size. In practice, while one c…
Recurrent neural networks are powerful models for processing sequential data, but they are generally plagued by vanishing and exploding gradient problems. Unitary recurrent neural networks (uRNNs), which use unitary recurrence matrices, have recently been proposed as a means to avoid these issues. However, in previous …
We extend manifold capacity to nonlinear neural representations with contextual information.
problem Efficient processing of information through neural representations.
method Theoretical framework leveraging latent directions in input space related to contextual information.
result Derivation of an exact formula for context-dependent manifold capacity.
gLSTM improves graph neural networks by increasing storage capacity to prevent over-squashing.
problem Over-squashing in GNNs collapses information from a large receptive field into a single vector, creating an information bottleneck.
method Introduced a new synthetic task to measure over-squashing and adapted ideas from sequence modeling to develop gLSTM, a novel GNN architecture with improved capacity.
result gLSTM architecture demonstrates strong performance on synthetic and real-world graph benchmarks, mitigating over-squashing.
Millimeter Wave (MmWave) communication is one of the key technology of the fifth generation (5G) wireless systems to achieve the expected 1000x data rate. With large bandwidth at mmWave band, the link capacity between users and base stations (BS) can be much higher compared to sub-6GHz wireless systems. Meanwhile, due …
New algorithm for shareable arms with load-dependent rewards in stochastic bandits.
problem Learning optimal play strategy with shareable finite-capacity arms in stochastic bandits.
method Developed a capacity estimator and online learning algorithm for MP-MAB with shareable arms.
result Regret upper bound matches the lower bound, validating the algorithm's performance.
Deep imagination optimizes decision-making in large trees with limited resources.
problem Optimal planning in large decision trees with limited resources and time.
method Analytical solutions and numerical analysis of sampling capacity allocation.
result Optimal policy is to allocate few samples per level for deep exploration, favoring depth over breadth.
Study online learning with delays and capacity constraints, achieving optimal regret bounds.
problem Online learning with delays and capacity constraints.
method Novel scheduling and preemptive techniques, matching upper and lower bounds.
result Achieves optimal regret bounds across all capacity levels.
Sharp limits found for storing and retrieving input-output associations in linear associative memories.
problem Understanding the fundamental limits of storing and retrieving input-output associations in neural networks.
method Study of a minimal linear associative memory model, introducing a decoupled model and using statistical physics to characterize storage capacity.
result Linear associative memory can store up to 1/2 log(p) associations, providing a sharp statistical-physics characterization.
Study characterizes hulls and capacities on Riemannian manifolds, proving isoperimetric inequalities.
problem Characterizing hulls and capacities on Riemannian manifolds.
method Investigates strictly outward minimising hulls and uses p-capacities to recover their areas.
result Sharp isoperimetric inequality on complete noncompact manifolds with nonnegative Ricci curvature.
Enhances sequence memory capacity in neural networks.
problem Limited sequence capacity in Hopfield-like neural networks.
method Introducing a nonlinear interaction term and a generalized pseudoinverse rule.
result Significantly increased sequence capacity with novel scaling laws.
The paper tackles imbalanced classification under operational constraints, proposing a framework to maximize sensitivity.
problem Detecting minority class observations under severe class imbalance and operational constraints.
method Formal classification framework under capacity constraints, maximizing sensitivity while respecting a user-defined label limit.
result The optimal classifier under capacity constraints is equivalent to the Bayes classifier with reweighted prior probabilities.
Researchers analyze neural process architectures and their representational capacities.
problem Understanding what functions can be represented by different neural process architectures.
method Analyzing four types of neural process architectures: CNPs, ANPs, TNPs, and their latent variants.
result Prove these architectures form a strict hierarchy and characterize their representational capabilities.
Following the recent work on capacity allocation, we formulate the conjecture that the shattering problem in deep neural networks can only be avoided if the capacity propagation through layers has a non-degenerate continuous limit when the number of layers tends to infinity. This allows us to study a number of commonly…
While the channel capacity reflects a theoretical upper bound on the achievable information transmission rate in the limit of infinitely many bits, it does not characterise the information transfer of a given encoding routine with finitely many bits. In this note, we characterise the quality of a code (i. e. a given en…
Wide neural networks can degrade performance, contrary to conventional wisdom.
problem Understanding the limitations of increasing network width in neural networks.
method Using Deep Gaussian Processes to decouple capacity and width, analyzing their effects on representational power and non-Gaussianity.
result Wide neural networks can become less adaptable and more Gaussian, leading to performance degradation.
Study the relative volume function on AH manifolds and its applications.
problem Characterize the height of geodesic defining functions and capacity of balls.
method Define and analyze the relative volume function, proving its boundedness and regularity.
result Uniformly bounded relative volume function at infinity, bound dependent only on dimension.
RAF model explains neural networks' dual rule learning and fact memorization.
problem Understanding how neural networks learn rules and memorize facts simultaneously.
method Introduces the Rules-and-Facts (RAF) model to bridge generalization and memorization.
result Characterizes conditions for simultaneous rule learning and fact memorization in neural networks.
Proposes unbiased estimators for training mixture of experts models.
problem Efficiently training large-scale mixture of experts models on modern hardware.
method Two unbiased estimators based on principled stochastic assignment procedures.
result Both estimators are more effective and robust than biased alternatives.
Estimates on Einstein manifolds improve Brownian motion behavior and curvature limits.
problem Improving estimates on Einstein manifolds for Brownian motion behavior.
method Generalizing Benjamini-Pemantle-Peres estimate to manifolds with Ricci curvature bounds.
result Sharp estimates for Brownian motion on high curvature parts of Ricci-flat manifolds.
Deep neural networks with more parameters and FLOPs have higher capacity and generalize better to diverse domains. But to be deployed on edge devices, the model's complexity has to be constrained due to limited compute resource. In this work, we propose a method to improve the model capacity without increasing inferenc…
Semi-supervised node classification in attributed graphs, i.e., graphs with node features, involves learning to classify unlabeled nodes given a partially labeled graph. Label predictions are made by jointly modeling the node and its' neighborhood features. State-of-the-art models for node classification on such attrib…
The capacity of a neural network to absorb information is limited by its number of parameters. Conditional computation, where parts of the network are active on a per-example basis, has been proposed in theory as a way of dramatically increasing model capacity without a proportional increase in computation. In practice…
It has been known for a long time that the classical spherical perceptrons can be used as storage memories. Seminal work of Gardner, \cite{Gar88}, started an analytical study of perceptrons storage abilities. Many of the Gardner's predictions obtained through statistical mechanics tools have been rigorously justified. …
New insights into experience replay in RL algorithms.
problem Understanding the impact of replay capacity and replay ratio in Q-learning.
method Systematic and extensive analysis of experience replay in Q-learning methods, focusing on replay capacity and replay ratio.
result Greater replay capacity significantly improves performance for certain algorithms, while other techniques offer limited benefit.
New Weyl's laws discovered for compact spaces with Ricci curvature bounds.
problem Understanding growth rates of eigenvalues in compact spaces with Ricci curvature constraints.
method Developed new properties of α-Grushin halfplanes and analyzed singular sets of null capacities. result Established Weyl's laws with power growth and logarithmic corrections for compact spaces.
Study homeomorphism groups of ordinals, proving strong distortion and normal generators.
problem Understanding algebraic and geometric properties of homeomorphism groups of ordinals.
method Analyzing successor ordinals with connections to permutation groups and manifolds.
result Proves strong distortion and normal generators for homeomorphism groups of ordinals.
This paper finds the noise threshold for learning Gaussian mixture models equals channel capacity.
problem Learning Gaussian mixture models with noisy data.
method Bayesian formulation with uniformly distributed centers on a sphere, analyzing the large system limit.
result The maximal noise level σ2 for which GMM learning is as easy as labeled observations is the channel capacity. Transformers learn to cluster Gaussian mixtures as well as the EM algorithm.
problem Learning guarantees of Transformers in multi-class clustering of Gaussian mixtures.
method Developed a theory connecting Transformer's Softmax Attention layers to the EM algorithm's workflow.
result Transformers achieve minimax optimal rate for clustering Gaussian mixtures with sufficient training samples and initialization.
We study various capacities on compact Kähler manifolds which generalize the Bedford-Taylor Monge-Ampère capacity. We then use these capacities to study the existence and the regularity of solutions of complex Monge-Ampère equations.
This paper provides theoretical insights into why and how deep learning can generalize well, despite its large capacity, complexity, possible algorithmic instability, nonrobustness, and sharp minima, responding to an open question in the literature. We also discuss approaches to provide non-vacuous generalization guara…
Solves a discrete logarithmic Minkowski problem for electrostatic p-capacity.
problem Characterize measures generated by electrostatic p-capacity.
method Solves the discrete logarithmic Minkowski problem for 1 < p < n.
result Solves the discrete logarithmic Minkowski problem for measures in general position.
CapOptix uses options theory to price capacity in electricity markets.
problem Traditional capacity market designs fail to account for risk and price shocks.
method Interprets capacity commitments as reliability options and uses Markov Regime Switching Process.
result CapOptix provides more accurate pricing of capacity premia compared to existing mechanisms.
In this article, we propose the notion of the general p-affine capacity and prove some basic properties for the general p-affine capacity, such as affine invariance and monotonicity. The newly proposed general p-affine capacity is compared with several classical geometric quantities, e.g., the volume, the p-var…
While symplectic manifolds have no local invariants, they do admit many global numerical invariants. Prominent among them are the so-called symplectic capacities. Different capacities are defined in different ways, and so relations between capacities often lead to surprising relations between different aspects of sympl…
DNPUs improve neural network performance with high-capacity nanoelectronic nodes.
problem Limited performance of single DNPUs in solving complex classification problems.
method Developed DNPUs as high-capacity neurons and implemented multi-DNPU networks.
result Feed-forward DNPU networks improve single DNPU performance from 77% to 94% test accuracy.