New approach finds minimum width for deep, narrow MLPs.
problem Finding the minimum width for deep, narrow MLPs to approximate continuous functions.
method Proposes a framework to simplify finding minimum width into determining a geometrical function w(dx,dy) based on input and output dimensions. result Proves that w(dx,dy) equals the optimal minimum width for deep, narrow MLPs to achieve universality. Narrow neural networks have unbounded decision regions.
problem Understanding decision regions of narrow neural networks.
method Analyzing decision regions of neural networks with width ≤ input dimension.
result All connected components of decision regions are unbounded.
Machine learning models predict crash rates on narrow lanes.
problem Impact of narrow lanes on arterial road vehicle crashes.
method Applied random forest and least squares boosting machine learning algorithms to crash data.
result Random forest model identified as best for studying narrow lanes' safety impact.
Deep narrow networks can approximate any continuous function.
problem Approximating continuous functions with neural networks of bounded width and arbitrary depth.
method Showed neural networks of arbitrary depth, width n+m+2, and activation function ρ is dense in C(K;Rm) for K⊆Rn with K compact. result Neural networks of bounded width and arbitrary depth can approximate any continuous function.
Wide networks avoid bad local minima, proving phase transition.
problem Understanding the optimization landscape of neural networks.
method Proving phase transitions in loss surfaces of wide and narrow networks.
result Phase transition from narrow to wide networks: no bad local minima.
Study proves deep narrow RNNs can approximate any function, with minimum width independent of data length.
problem Proving universality of deep narrow RNNs with bounded widths.
method Analyzing RNNs as dynamical systems, proving universality for deep narrow structures with specific widths.
result Minimum width for universality of deep narrow RNNs is independent of data length.
Paper calculates topological complexity of robot movement in narrow aisles.
problem Determining minimum number of scenarios for robot movement in a narrow strip.
method Examined cohomology ring of ordered configuration space to find lower bound.
result Lower bound for minimum number of cases in robot movement program.
Improves matrix multiplication throughput for asymmetric bit-width operands.
problem Matrix multiplications between asymmetric bit-width operands, especially 8- and 4-bit, are not efficiently handled by existing SIMD instructions.
method Proposes a new SIMD matrix multiplication instruction that uses mixed precision on inputs (8- and 4-bit) and accumulates into 16-bit output, improving throughput.
result Offers 2x improvement in throughput compared to existing symmetric-operand-size instructions, with negligible overflow.
Complex-valued neural networks can approximate any continuous function with bounded widths and depths.
problem Approximating continuous functions with complex-valued neural networks of bounded widths and depths.
method Analyzing activation functions and proving universality for complex-valued networks.
result Deep narrow complex-valued networks are universal if and only if their activation function is neither holomorphic, nor antiholomorphic, nor R-affine. We show that deep narrow Boltzmann machines are universal approximators of probability distributions on the activities of their visible units, provided they have sufficiently many hidden layers, each containing the same number of units as the visible layer. We show that, within certain parameter domains, deep Boltzmann…
New findings show depth is more important than width in neural networks.
problem Understanding the role of width and depth in neural networks.
method Constructed networks with bounded weights and width at most d+2, showing depth plays a more significant role.
result Depth is more important than width in the expressive power of neural networks.
Wider networks learn more interpretable features and improve performance during fine-tuning.
problem Transferability of learned features between tasks and the effect of network width on feature learning.
method Activation atlases to visualize and analyze features learned by wide and narrow networks.
result The hidden state of a wide network contains more information about the inputs than a narrow network, leading to improved performance during fine-tuning.
Wide neural networks with narrow bottlenecks behave like deep Gaussian processes.
problem Understanding the behavior of neural networks with narrow layers in the wide limit.
method Analyzing the wide limit of BNNs with narrow bottlenecks, showing they behave like a composition of GPs.
result Wide neural networks with narrow bottlenecks form a composition of GPs, termed a bottleneck NNGP.
Wide neural networks can degrade performance, contrary to conventional wisdom.
problem Understanding the limitations of increasing network width in neural networks.
method Using Deep Gaussian Processes to decouple capacity and width, analyzing their effects on representational power and non-Gaussianity.
result Wide neural networks can become less adaptable and more Gaussian, leading to performance degradation.
Gradient descent finds global min in wide deep linear networks.
problem Optimizing deep linear neural networks with limited width.
method Proved convergence rate for gradient descent with wide layers.
result Gradient descent converges linearly to global min in wide layers.
Study compares random and learned features in deep Bayesian linear models.
problem Understanding how feature learning affects generalization in deep learning.
method Comparing deep random feature models to deep networks with trained layers.
result Random feature models can display double-descent behavior, while deep networks do not.
We generalize recent theoretical work on the minimal number of layers of narrow deep belief networks that can approximate any probability distribution on the states of their visible units arbitrarily well. We relax the setting of binary units (Sutskever and Hinton, 2008; Le Roux and Bengio, 2008, 2010; Montúfar and Ay,…
Empirical study compares wide neural networks to kernel methods, resolving open questions.
problem Understanding the relationship between wide neural networks and kernel methods.
method Large-scale empirical study using various neural network architectures and kernel methods.
result Wide neural networks outperform fully-connected finite-width networks in some cases, but underperform convolutional finite-width networks.
New framework analyzes deep learning optimization with finite width networks, revealing generalization gaps and excess risks.
problem Analyzing generalization error of deep learning with finite width networks.
method Formulating neural network training as transportation map estimation and analyzing via infinite dimensional Langevin dynamics.
result Achieves fast learning rate and minimax optimal rates for classification and regression problems.
This paper considers the generation of prediction intervals (PIs) by neural networks for quantifying uncertainty in regression tasks. It is axiomatic that high-quality PIs should be as narrow as possible, whilst capturing a specified portion of data. We derive a loss function directly from this axiom that requires no d…
Project aims to improve time series prediction intervals.
problem Difficulty in computing joint prediction regions for time series data.
method Wolf and Wunderli's method applied with bootstrapping and novel standard error estimation.
result Empirical evidence on the effectiveness of the method.
A new method reduces memory requirements for Graph Transformers by sparsely training a network.
problem Quadratic memory complexity in Graph Transformers limits their scalability to large graphs.
method Spexphormer: trains a narrow network on augmented graph, then uses only active connections in a wider network.
result Spexphormer achieves good performance with drastically reduced memory requirements.
Statistical physics explains deep learning's feature learning capacity.
problem Understanding neural networks' ability to learn complex features.
method Study of a multi-layer perceptron in the interpolation regime.
result Optimal learning requires specialization across layers and neurons.
New method predicts aphasia severity with narrower uncertainty intervals.
problem Predicting aphasia severity in stroke patients using neuroimages.
method Sparse heteroscedastic Bayesian high-dimensional regression with H-PROBE algorithm.
result H-PROBE provides narrower prediction intervals for aphasia severity.
Deep ResNets with single-neuron hidden layers can approximate any function.
problem The challenge of universal approximation by deep neural networks.
method A ResNet architecture with one neuron per hidden layer in each module.
result ResNet with one-neuron hidden layers is a universal approximator.
New insights into neural network feature learning through multi-step gradient descent.
problem Understanding feature learning in two-layer neural networks with limited width.
method Characterization of feature learning through two steps of gradient descent with specific step sizes.
result The second step of gradient descent reveals multiple learned directions, not limited to a single direction as in the first step.
ResRep prunes CNNs without losing accuracy by separating remembering and forgetting.
problem Pruning CNNs to reduce FLOPs without sacrificing accuracy.
method Decoupling remembering and forgetting in CNNs, using SGD for remembering and a novel update rule for forgetting.
result Achieved lossless pruning with high compression ratio (76.15% accuracy on ImageNet with 45% FLOPs reduction).
Deep neural networks can classify any dataset with exact solutions.
problem Classifying any dataset with multilayer perceptrons.
method Constructive analysis of multilayer perceptrons with exact solutions.
result Explicit constructions of MLPs for classification and function approximation.
VLM judges rank well but score poorly; task difficulty and annotation quality affect interval width.
problem VLMs as judges lack reliability indicators in multimodal evaluations.
method Conformal prediction using score-token log-probabilities.
result Evaluation uncertainty is task-dependent, affecting interval width and reliability.
Investigates the impact of narrow banking on macroeconomics.
problem The risks and benefits of a full reserve requirement on demand deposits.
method Extended Goodwin-Keen model with time deposits and central bank reserves; numerical examples.
result Narrow banking does not reduce economic growth but improves financial stability.
RAGIC predicts stock intervals with risk considerations, improving prediction accuracy and coverage.
problem Limited success in predicting stock market outcomes due to stochastic nature and risk oversight.
method RAGIC uses a GAN with a risk module and temporal module to generate risk-sensitive stock intervals.
result RAGIC achieves a consistent 95% coverage with narrow interval widths, balancing accuracy and risk.
Study challenges the Gaussian pre-activations assumption in neural networks.
problem Challenges the assumption that pre-activations are Gaussian in neural networks.
method Constructs pairs of activation functions and initialization distributions to ensure Gaussian pre-activations.
result Discovered constraints for ensuring Gaussian pre-activations in neural networks.
Study of infinitely deep but narrow neural networks using NTK theory.
problem Analyzing the role of depth in deep learning with overparameterized networks.
method Infinite-depth limit analysis of MLP and CNN using Neural Tangent Kernel (NTK) theory.
result Established trainability guarantee for infinitely deep but narrow neural networks.
Deep and narrow neural networks can collapse to incorrect states.
problem The collapse of deep and narrow neural networks to incorrect states.
method Numerical and theoretical analysis of deep and narrow neural networks with ReLU activation.
result Deep and narrow neural networks can converge to erroneous mean or median states with high probability.
CIR method constructs efficient prediction intervals with guaranteed coverage.
problem Efficiently constructing near-minimal prediction intervals with guaranteed coverage.
method Conditional Interquantile Regression (CIR) and CIR+ (enhanced version).
result Optimal balance between predictive accuracy and computational efficiency.
This paper presents a method to automatically generate high-quality prediction intervals for neural networks.
problem Accurate uncertainty quantification for deep learning models in real-world applications.
method Dual neural network approach with a novel loss function to balance prediction interval width and coverage.
result Our method produces significantly narrower prediction intervals with higher probability coverage compared to state-of-the-art methods.
Study predicts risk of true-lumen narrowing after ATAAD surgery using CT data.
problem Early post-surgery risk assessment for aortic dissection patients.
method Retrospective study with CT data, derived cross-sectional shapes, form factor (FF) for morphology assessment, linear discriminant analysis (LDA) for risk classification, LOPO-CV for prediction.
result Machine-learning model accurately predicts risk for all high-risk patients and low-risk patients, potentially reducing hospital visits.
Study on ion travel time on curved surfaces.
problem Mean first passage time of ion on curved surfaces.
method Layer potential argument and microlocal analysis.
result Derivation of mean first passage time and spatial average.
New findings on hidden symmetries in ReLU networks.
problem Understanding the redundancy and symmetries in ReLU network parameter space.
method Analyzing parameter settings and function classes for various network architectures.
result For certain network architectures, there are no hidden symmetries.
Light pillars over rippled water appear parallel due to projection geometry.
problem Light pillars over rippled water appear parallel.
method Developing a geometric optics model
result The phenomenon is explained using the specular reflection rule, projection geometry, and physics of surface slopes.
Random Forests provide interpretable prediction intervals with theoretical guarantees.
problem Lack of uncertainty estimates in machine learning point predictions.
method Out-of-Bag procedure for generating parametric and non-parametric prediction intervals.
result Proposed prediction intervals deliver correct coverage rates and narrow lengths.
Emergent misalignment is influenced by training dynamics, model priors, and data.
problem Emergent misalignment in models
method Exploring training dynamics, model priors, and data
result Activation deltas before and after narrow fine-tuning correlate with their similarities when measured with the last prompt-token activations.
An algorithm calculates Gabai width for thousands of knots.
problem Calculating Gabai width for many knots.
method Algorithmic definition of Wirtinger width leading to efficient Gabai width bounds.
result Proved Wirtinger width equals Gabai width for knots.
Study geodesic orbits on noncompact curved spaces, proving their distribution and counting.
problem Counting and equidistribution of periodic orbits on noncompact manifolds.
method Proved equidistribution in narrow topology, deduced exact asymptotic counting.
result Exact asymptotic counting of periodic orbits on noncompact manifolds.
Study extends knot width concept to higher dimensions, finding large widths possible.
problem Defining and measuring knot width in higher dimensions.
method Extended classical width definition to smooth codimension 2 knots in higher dimensions.
result Arbitrarily large widths of knots possible in each dimension.
Empirical study compares finite- and infinite-width BNNs, revealing performance differences under model mismatch.
problem Comparing BNNs with different widths due to conflicting model properties and inference intractability.
method Empirical comparison of finite- and infinite-width BNNs, analyzing performance under model mismatch.
result Increasing width can hurt BNN performance when the model is mis-specified, and finite-width BNNs generalize better under model mismatch.
2-width of 3-manifolds is bounded by 2 but embeddings can have infinite 2-width.
problem Defining and analyzing the 2-width of 3-manifolds and their embeddings.
method Generalizing width concept to 3-manifolds and embeddings, showing bounds and divergences.
result Embeddings of 3-manifolds can have arbitrarily large 2-width, challenging classical width concepts.
The isospectral problem for p-widths is solved using Zoll metrics on S^2.
problem Determine if a Riemannian manifold is uniquely determined by its p-widths.
method Construct counterexamples on S^2 using Zoll metrics and properties of geodesic p-widths.
result Many counterexamples exist on S^2, showing uniqueness is not guaranteed.