This paper analyzes deep Stable neural networks, showing convergence rates under different growth settings.
problem Analyzing the behavior of deep Stable neural networks as width increases.
method Large-width asymptotic analysis and convergence rates for fully connected feed-forward deep Stable NNs.
result The rescaled deep Stable NN converges weakly to a Stable SP under joint growth, with sup-norm convergence rates established.
Method reconstructs aneurysm growth history from patient parameters using physics-informed autoencoder.
problem Predicting arterial aneurysm rupture due to inaccessible growth time series.
method Physics-informed autoencoder combined with neural network for mapping patient parameters to aneurysm growth time history.
result Incorporating physical model constraints improves time series reconstruction, especially in noisy data.
Despite the phenomenal success of deep learning in recent years, there remains a gap in understanding the fundamental mechanics of neural nets. More research is focussed on handcrafting complex and larger networks, and the design decisions are often ad-hoc and based on intuition. Some recent research has aimed to demys…
In recent years there has been an increased interest in statistical analysis of data with multiple types of relations among a set of entities. Such multi-relational data can be represented as multi-layer graphs where the set of vertices represents the entities and multiple types of edges represent the different relatio…
A new method, Multi-Label Deep Forest, tackles multi-label learning problems.
problem Leveraging label correlations in multi-label learning models.
method Designs a deep forest framework with two mechanisms: measure-aware feature reuse and measure-aware layer growth.
result Outperforms compared methods on six measures across benchmark datasets.
Study shows how large neural networks avoid overfitting through decoupling of feature learning and complexity growth.
problem Understanding inductive bias and generalization in large neural networks.
method Dynamical mean field theory applied to large two-layer networks.
result Training dynamics of large networks exhibit a separation of timescales, decoupling feature learning and overfitting.
Deep neural networks converge to Gaussian mixtures as layer width increases.
problem Understanding the distribution of outputs from deep neural networks.
method Proof and experiments with a simple model showing the convergence of neural network outputs to Gaussian mixtures.
result Neural networks converge to Gaussian mixtures as the width of the last hidden layer increases.
Upper bounds on fixed points in PWL neural networks with hyperplane analysis.
problem Analyzing the number of fixed points in neural networks with PWL activation.
method Hyperplane arrangements to bound the number of fixed points.
result Upper bounds on the number of fixed points for PWL networks, showing exponential growth in layers.
New theory explains signal propagation in normalization-free transformers.
problem Understanding signal propagation in normalization-free transformers.
method Deriving recurrence relations for activation statistics and APJNs across layers.
result Transformers with elementwise tanh-like nonlinearities exhibit subcritical signal propagation.
Proposes QEP to mitigate quantization error propagation in layer-wise post-training quantization.
problem Growth of quantization errors across layers degrades performance, especially in low-bit regimes.
method Quantization Error Propagation (QEP) framework that explicitly propagates and compensates for quantization errors.
result QEP-enhanced layer-wise PTQ achieves substantially higher accuracy, especially in low-bit regimes.
Enhances neural networks in hyperbolic space for better data structure capture.
problem Capturing hierarchical data structures efficiently.
method Unified hyperbolic model for neural network components.
result Superior parameter efficiency and outperformance over Euclidean methods.
LSTM predicts COVID-19 growth patterns from global data.
problem Forecasting the spread of COVID-19 using big data and machine learning.
method Used multivariate long short-term memory (LSTM) to learn correlations over time.
result LSTM outperformed RNN in predicting COVID-19 growth with lower validation error.
JFR-rg model explains Japan's stable debt despite high interest rates and low growth.
problem Understanding Japan's stable government debt despite high interest rates and low growth.
method Formalizes financial repression channels through JFR-rg model, incorporating financial repression bias and exchange-rate channel.
result Identifies Normalization Trap and Captive Financial System Parameter, showing debt dynamics under financial repression.
Random Transformers behave like polynomial models in ICL with asymptotic growth.
problem Understanding in-context learning capabilities of pretrained Transformers.
method Asymptotic analysis of a random Transformer with a fixed first layer and a trained second layer, considering growth in context length, input dimension, hidden dimension, and training parameters.
result The random Transformer's ICL error is equivalent to a finite-degree Hermite polynomial model.
Recent DNN pruning algorithms have succeeded in reducing the number of parameters in fully connected layers, often with little or no drop in classification accuracy. However, most of the existing pruning schemes either have to be applied during training or require a costly retraining procedure after pruning to regain c…
Study identifies stable configurations of intertwined threads with repulsive interactions.
problem Stable configurations of entangled systems with repulsive interactions.
method Analysis of steepest descent flow of an energy functional.
result Existence and uniqueness of stable configuration of two layers drifting apart at t1/3 rate. We propose a neural architecture search (NAS) algorithm, Petridish, to iteratively add shortcut connections to existing network layers. The added shortcut connections effectively perform gradient boosting on the augmented layers. The proposed algorithm is motivated by the feature selection algorithm forward stage-wise …
We demonstrate the possibility of what we call sparse learning: accelerated training of deep neural networks that maintain sparse weights throughout training while achieving dense performance levels. We accomplish this by developing sparse momentum, an algorithm which uses exponentially smoothed gradients (momentum) to…
One LN layer stabilizes neural network extrapolation.
problem Understanding neural networks' extrapolation behavior.
method Applied NTK theory to infinitely-wide networks.
result Adding one LN layer stabilizes network outputs.
SGD efficiently learns the XOR function with near-optimal sample complexity.
problem Learning the XOR function with a 2-layer neural network.
method Minibatch SGD on a 2-layer neural network with ReLU activations, focusing on signal-finding and signal-heavy phases.
result Achieves population error o(1) with dextpolylog(d) samples. New scaling laws explain deep learning performance growth.
problem Understanding neural network performance growth.
method Analyzed entire training dynamics of various architectures.
result Identified two dynamical scaling laws.
Study growth patterns in random networks using i.i.d. perturbations.
problem Understanding the growth of affine regions in random piecewise-linear networks.
method Analyzes a random compositional model with i.i.d. perturbations of the tent map, proving submultiplicative pressure and using finite-state defect process for upper-tail lower bounds.
result Proves the existence of a submultiplicative pressure for \(N_n\) and gives exponential upper bounds for \(n^{-1}\log N_n\).
A modular cash-overlay rule for allocating between a fixed growth-defensive risky sleeve and interest-bearing cash.
problem Drawdown control
method Continuous cash-overlay filters
result Earnings an 18.83% CAGR versus 16.62% for 100% R
Sharp stability threshold found for deep residual architectures.
problem Ensuring stable training and inference in deep residual networks.
method Sublinear-growth principle and optimal-control analysis.
result Stable training condition: input-magnitude exponent q ≤ 1.
Depth is a key component of Deep Neural Networks (DNNs), however, designing depth is heuristic and requires many human efforts. We propose AutoGrow to automate depth discovery in DNNs: starting from a shallow seed architecture, AutoGrow grows new layers if the growth improves the accuracy; otherwise, stops growing and …
This paper explains how low-precision arithmetic causes loss spikes in deep learning models.
problem Loss spikes during long-term training of deep neural networks.
method Analyzes the impact of floating-point precision limits on gradient updates and feature means.
result Numerical Feature Inflation (NFI) explains loss spikes and rapid parameter norm growth.
Deep Neural Networks (DNNs) have revolutionized numerous applications, but the demand for ever more performance remains unabated. Scaling DNN computations to larger clusters is generally done by distributing tasks in batch mode using methods such as distributed synchronous SGD. Among the issues with this approach is th…
Long short-term memory (LSTM) has been widely used for sequential data modeling. Researchers have increased LSTM depth by stacking LSTM cells to improve performance. This incurs model redundancy, increases run-time delay, and makes the LSTMs more prone to overfitting. To address these problems, we propose a hidden-laye…
Catapult phase in neural nets shows exponential loss growth before quick decrease.
problem Understanding phase transitions in neural networks during training.
method Analyzing weight norm and loss behavior for super-critical learning rates.
result Proven existence of catapult phase in quadratic models and two-layer nets.
Gradient descent trains both layers of a ReLU network to fit a linear model.
problem Training dynamics of a ReLU network to fit a linear target function.
method Jointly training both layers of a one-hidden-layer ReLU network in a realizable setting with Gaussian inputs and labels.
result Gradient descent from a small random initialization converges to a global minimizer at a linear rate with optimal sample complexity.
NeuroFabric proposes a method to optimize sparse network training topologies.
problem Long training times in deep neural networks due to high memory and compute requirements.
method Developed a new sparse neural network initialization scheme and evaluated various topologies.
result Identified a single optimal topology that maximizes accuracy across different datasets.
The universal approximation property of various machine learning models is currently only understood on a case-by-case basis, limiting the rapid development of new theoretically justified neural network architectures and blurring our understanding of our current models' potential. This paper works towards overcoming th…
Abstract Coxeter groups have growth rates that are Perron numbers.
problem Understanding growth rates of Coxeter groups.
method Defined a class of Coxeter groups, ∞--spanned, and analyzed their growth rates. result For ∞--spanned Coxeter groups, geodesic growth rate strictly dominates word growth rate and appears to be a Perron number. Study of deep Stable neural networks with various activation functions.
problem Characterizing the infinitely wide limits of deep Stable neural networks.
method Investigation of large-width properties of deep Stable NNs with a generalized central limit theorem for heavy tails.
result Extension of characterization to a broader class of activation functions, including sub-linear, asymptotically linear, and super-linear functions.
Historical economic growth in countries of the former USSR is analysed. It is shown that Unified Growth Theory is contradicted by the data, which were used, but not analysed, during the formulation of this theory. Unified Growth Theory does not explain the mechanism of economic growth. It explains the mechanism of Malt…
The Unified Growth Theory is a puzzling collection of myths based on illusions created by hyperbolic distributions. Some of these myths are discussed. The examination of data shows that the three stages of growth (Malthusian Regime, Post-Malthusian Regime and Modern Growth Regime) did not exist and that Industrial Revo…
Historical economic growth in Asia (excluding Japan) is analysed. It is shown that Unified Growth Theory is contradicted by the data, which were used (but not analysed) during the formulation of this theory. Unified Growth Theory does not explain the mechanism of economic growth. It explains the mechanism of Malthusian…
Study confined subgroups in groups with contracting elements, showing their growth rate is strictly greater than half of the ambient growth rate.
problem Understanding the growth rate of confined subgroups in groups with contracting elements.
method Through boundary actions, analyzing the Hopf decomposition and quotient growth.
result Confined subgroups have a growth rate strictly greater than half of the ambient growth rate.
Growth rate of the world Growth Domestic Product (GDP) is analysed to determine possible pathways of the future economic growth. The analysis is based on using the latest data of the World Bank and it reveals that the growth rate between 1960 and 2014 was following a trajectory approaching asymptotically a constant val…
Historical economic growth in Latin America is analysed using the data of Maddison. Unified Growth Theory is found to be contradicted by these data in the same way as it is contradicted by the economic growth in Africa, Asia, former USSR, Western Europe, Eastern Europe and by the world economic growth. Paradoxically, U…
Growth rates of geodesics on modular orbifolds are studied.
problem Understanding growth rates of geodesics on modular orbifolds.
method Exhaustion of modular orbifold by compact subsurfaces, analysis of low lying geodesics and reciprocal geodesics.
result Growth rates of low lying geodesics and reciprocal geodesics converge to the full set's growth rate.
Data describing historical economic growth are analysed. Included in the analysis is the world and regional economic growth. The analysis demonstrates that historical economic growth had a natural tendency to follow hyperbolic distributions. Parameters describing hyperbolic distributions have been determined. A search …
New multigrid approach reduces CNN parameters by focusing on structured convolutions.
problem Redundancy in standard CNNs leads to high parameter count.
method Replace standard convolutions with structured multilevel convolutions.
result Linearly proportional number of parameters to network width, no loss in accuracy.
The study provides volume growth estimates for specific types of manifolds.
problem Estimating volume growth for Ricci solitons and quasi-Einstein manifolds.
method Similar to classical results, the study proves volume growth estimates for gradient Ricci solitons and quasi-Einstein manifolds.
result Sharp volume growth estimates for gradient shrinking Ricci solitons and upper bound volume growth estimates for quasi-Einstein manifolds.
Residual networks' depth is mathematically equivalent to expanding an implicit ensemble size.
problem Understanding why deep residual networks are effective.
method Formal analysis of residual networks as ensembles of shallow models.
result Increasing network depth is equivalent to expanding the size of an implicit ensemble, revealing a hierarchical structure.
Study growth rates of subgroups in groups with a constricting element.
problem Understanding growth rates of subgroups in groups with a constricting element.
method Examining the spectrum of relative and quotient exponential growth rates of quasi-convex subgroups.
result Determine when growth rates of subgroups are strictly smaller or coincide with the group's growth rate.
We present a quantitative characterisation of the fluctuations of the annualized growth rate of the real US GDP per capita growth at many scales, using a wavelet transform analysis of two data sets, quarterly data from 1947 to 2015 and annual data from 1800 to 2010. Our main finding is that the distribution of GDP grow…
Paper reviews and proves volume growth estimates for different types of gradient Ricci solitons.
problem Estimating volume growth for gradient Ricci solitons.
method Survey and prove new volume growth estimates.
result New volume growth estimates for expanding gradient Ricci solitons.