Stochastic regularisation is an important weapon in the arsenal of a deep learning practitioner. However, despite recent theoretical advances, our understanding of how noise influences signal propagation in deep neural networks remains limited. By extending recent work based on mean field theory, we develop a new frame…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,657 papers · 148 categories
Trend · papers per month
48 results for “signal propagation”
Field theory for optimal signal propagation in ResNetscond-mat.dis-nn
Field theory explains optimal scaling in ResNets for signal propagation.
problem Understanding optimal scaling parameter for ResNet performance.
method Finite-size field theory for ResNets to study signal propagation and scaling.
result Analytical expressions for optimal scaling parameter, independent of other hyperparameters.
Bayesian neural networks (BNNs) have developed into useful tools for probabilistic modelling due to recent advances in variational inference enabling large scale BNNs. However, BNNs remain brittle and hard to train, especially: (1) when using deep architectures consisting of many hidden layers and (2) in situations wit…
Study eigenvalues and eigenvectors in neural networks, focusing on signal propagation.
problem Characterize signal eigenvalues and eigenvectors in neural networks.
method Characterizes signal eigenvalues and eigenvectors for a nonlinear spiked covariance model.
result Provides precise quantitative characterizations of signal eigenvalues and eigenvectors in neural networks.
Beyond Signal Propagation: Is Feature Diversity Necessary in Deep Neural Network Initialization?cs.LG
New method shows random, diverse initializations are not essential for deep neural networks.
problem The necessity of random, diverse initializations in deep neural networks.
method Constructed a deep convolutional network with identical features by initializing weights to 0, enabling signal propagation and stable gradients.
result Random, diverse initializations are not necessary for training neural networks.
Deep vanilla transformers trained without shortcuts achieve similar performance to standard models.
problem Training deep vanilla transformers without shortcuts and normalizations.
method Parameter initializations, bias matrices, and location-dependent rescaling.
result Deep vanilla transformers can train at similar speeds and performance to standard models.
New theory explains signal propagation in normalization-free transformers.
problem Understanding signal propagation in normalization-free transformers.
method Deriving recurrence relations for activation statistics and APJNs across layers.
result Transformers with elementwise tanh-like nonlinearities exhibit subcritical signal propagation.
We find ways to make physical signals misclassified by computer vision models.
problem Vulnerability of signal classifiers to adversarial perturbations in physical signals.
method Solving PDE-constrained optimization problems to construct imperceptible perturbations.
result Effective and physically realizable adversarial perturbations can be computed for machine learning models.
Optimal liquidation strategy with price impact and signal exploitation.
problem Maximizing revenue-risk in a market with transient and temporary price impact.
method Infinite dimensional stochastic control approach, backward stochastic differential equation, operator-valued Riccati equation.
result Explicit expression for the optimal trading strategy.
Effective theory for Transformer initialization improves model performance.
problem Improving performance of Transformers at initialization.
method Effective-theory analysis of signal propagation in wide and deep Transformers.
result Particular width scalings of initialization and training hyperparameters.
New method improves ResNet performance without batch normalization.
problem Improving ResNet performance without batch normalization.
method Adapted Weight Standardization to maintain signal propagation.
result Highly performant ResNets achieve state-of-the-art performance on ImageNet.
Deep neural networks show some layers better align with data than others.
problem Understanding why some layers in deep neural networks better align with data.
method Introducing the Equilibrium Hypothesis to connect alignment pattern to signal propagation.
result The Equilibrium Hypothesis explains the ascent-descent pattern of alignment in deep neural networks.
Information Propagation Across Investor Types: Transfer Entropy Networks in the Korean Equity Marketq-fin.ST
Investor flows in Korean equity market transmit shared information, not private signals.
problem Whether investor flows transmit private information or only public signals.
method Transfer Entropy networks constructed from investor-type flows over
umNDates{} trading days.
result Investor flows transmit shared information, not private signals.
Signals are submanifolds; bounds on energy calculated.
problem Abstract theory of signal propagation.
method Energy inequalities and bounds calculated for specific signal spaces.
result Upper and lower bounds on energy derived for various signal configurations.
This paper proposes an alternating back-propagation algorithm for learning the generator network model. The model is a non-linear generalization of factor analysis. In this model, the mapping from the continuous latent factors to the observed signal is parametrized by a convolutional neural network. The alternating bac…
Network pruning is a promising avenue for compressing deep neural networks. A typical approach to pruning starts by training a model and then removing redundant parameters while minimizing the impact on what is learned. Alternatively, a recent approach shows that pruning can be done at initialization prior to training,…
Training recurrent neural networks (RNNs) on long sequence tasks is plagued with difficulties arising from the exponential explosion or vanishing of signals as they propagate forward or backward through the network. Many techniques have been proposed to ameliorate these issues, including various algorithmic and archite…
New weight initialisation for ICNNs accelerates learning and improves generalization.
problem Lack of effective initialisation strategies for ICNNs due to their unique weight and activation properties.
method Derived a principled weight initialisation by generalizing signal propagation theory for ICNNs with non-negative weights.
result Principled initialisation effectively accelerates learning and leads to better generalization in ICNNs.
The study finds that supply chain information from LLM embeddings improves stock returns predictions.
problem Predicting stock returns using textual information from annual reports.
method Combining LLM embeddings of annual reports with supply chain knowledge graph propagation.
result Network-augmented embeddings significantly predict stock returns with a Sharpe ratio of 0.86 and alpha of 7.27%.
NoProp learns neural networks without full back-propagation or forward-propagation.
problem Learning hierarchical representations in neural networks.
method NoProp independently learns each block to denoise a noisy target using local targets and back-propagation within the block.
result NoProp is a viable learning algorithm that is easy to use and computationally efficient.
Receiver algorithms which combine belief propagation (BP) with the mean field (MF) approximation are well-suited for inference of both continuous and discrete random variables. In wireless scenarios involving detection of multiple signals, the standard construction of the combined BP-MF framework includes the equalizat…
Wi-GATr learns to simulate wireless signals with high accuracy and speed.
problem Inaccurate wireless signal propagation models limit modern communication system design.
method Wi-GATr uses a Geometric Algebra Transformer to learn from scene primitives.
result Wi-GATr achieves more accurate predictions than existing methods.
In this study, we propose a novel deep neural network and its supervised learning method that uses a feedforward supervisory signal. The method is inspired by the human visual system and performs human-like association-based learning without any backward error propagation. The feedforward supervisory signal that produc…
Reducing the precision of weights and activation functions in neural network training, with minimal impact on performance, is essential for the deployment of these models in resource-constrained environments. We apply mean-field techniques to networks with quantized activations in order to evaluate the degree to which …
A new approach estimates propagators for trading risky assets.
problem Estimating price impact kernel from static data for optimal trading.
method Nonparametric estimation of propagator using offline reinforcement learning.
result Pessimistic trading strategy optimises execution costs under uncertainty.
Algorithm maximizes revenue-risk by estimating price impact kernel and optimizing control problems.
problem Maximizing revenue-risk in a risky asset liquidation with unknown price impact.
method Alternates exploration and exploitation phases, uses novel kernel estimation and stability results.
result Sublinear regret achieved with high probability.
This research explores principles of Lipschitz continuity in neural networks for robustness and generalization.
problem Ensuring robustness and generalization in neural networks, especially to small input perturbations and out-of-distribution data.
method Two complementary perspectives: internal (training dynamics) and external (frequency signal propagation).
result Advances in understanding the principles of Lipschitz continuity in neural networks.
This paper presents a novel approach for approximate integration over the uncertainty of noise and signal variances in Gaussian process (GP) regression. Our efficient and straightforward approach can also be applied to integration over input dependent noise variance (heteroscedasticity) and input dependent signal varia…
This paper analyzes Stochastic Depth regularization in ResNets.
problem Improving generalization in ResNets through regularization.
method Hybrid analysis combining perturbation and signal propagation.
result Principled guidelines for choosing survival rates in SD.
Paper proposes a new MIMO detection algorithm using Gaussian Mixture Expectation Propagation.
problem Challenges in MIMO detection due to interference and noise in high-order high-dimensional systems.
method The approach uses a Gaussian Mixture Model (GMM) approximation for Belief Propagation (BP) and Expectation Propagation (EP) messages to improve detection accuracy.
result The proposed algorithm outperforms state-of-the-art detection algorithms while maintaining low computational complexity.
Paper uses queue theory to model financial signals with relativistic delay.
problem Relativistic delay in financial trading signals.
method Modified M/M/G queue theory.
result Describes propagation of trading signals with finite velocity.
Two EP frameworks ensure integrable beliefs in Bayesian estimation problems.
problem Non-integrable beliefs in EP can lead to infeasible solutions.
method Proposes two EP frameworks to keep messages non-integrable.
result Ensures integrable beliefs in EP, even with non-integrable messages.
Optimal portfolio choice with cross-impact propagators, solving complex equations.
problem Maximizing revenue-risk in a continuous-time portfolio choice problem with cross-impact.
method Formulated as a maximization problem, solved explicitly using operator resolvents and stochastic Fredholm equations.
result Sufficient conditions for the absence of price manipulation, providing financial insights.
Many practical machine learning tasks employ very deep convolutional neural networks. Such large depths pose formidable computational challenges in training and operating the network. It is therefore important to understand how fast the energy contained in the propagated signals (a.k.a. feature maps) decays across laye…
Artificial neural networks are most commonly trained with the back-propagation algorithm, where the gradient for learning is provided by back-propagating the error, layer by layer, from the output layer to the hidden layers. A recently discovered method called feedback-alignment shows that the weights used for propagat…
Adversarial inference on tree models is possible with limited corruption, improving on Kesten-Stigum threshold.
problem Posterior inference on tree-structured graphical models in the presence of adversarial corruption.
method Dynamic programming via belief propagation, constrained adversarial corruption.
result Belief propagation can perform accurate inference with limited adversarial corruption.
We present a Bayesian method for feature selection in the presence of grouping information with sparsity on the between- and within group level. Instead of using a stochastic algorithm for parameter inference, we employ expectation propagation, which is a deterministic and fast algorithm. Available methods for feature …
Optimal trading strategy derived for nonlinear price impact models.
problem Optimal trading with nonlinear price impact induced by alpha signals.
method Variational approach, nonlinear Fredholm equation, iterative scheme.
result Existence and uniqueness of optimal trading strategy under monotonicity condition.
Light neural network detects modulation in noisy signals.
problem Efficiently detecting modulation in noisy signals.
method Light neural network architecture invariant to impairments.
result Network achieves accuracy under realistic impairments.
New method uncovers small but significant local activities in time-series data.
problem Reconstructing small but important local activities in time-series data.
method Neural state-space models with latent causal-effect disentanglement.
result Demonstrated proof-of-concept on reconstructing ectopic foci in cardiac electrical propagation.
Recent work in the domain of misinformation detection has leveraged rich signals in the text and user identities associated with content on social media. But text can be strategically manipulated and accounts reopened under different aliases, suggesting that these approaches are inherently brittle. In this work, we inv…
A simple gating mechanism improves deep learning convergence.
problem Vanishing or exploding gradients in deep networks.
method Introducing a zero-initialized parameter to each residual connection.
result Training deep networks (up to 120 layers) with fast convergence and better performance.
We consider the problem of joint modelling of metabolic signals and gene expression in systems biology applications. We propose an approach based on input-output factorial hidden Markov models and propose a structured variational inference approach to infer the structure and states of the model. We start from the class…
Topological Risk Parityq-fin.PM
TRP uses tree-based approach for market-neutral portfolios.
problem Creating non-binary, market-neutral portfolios with signed signals.
method Tree-based portfolio construction with minimum-spanning-tree and sector-anchored variants.
result TRP outperforms HRP in preserving signal direction and managing exposures.
Gaussian Belief Propagation (BP) algorithm is one of the most important distributed algorithms in signal processing and statistical learning involving Markov networks. It is well known that the algorithm correctly computes marginal density functions from a high dimensional joint density function over a Markov network i…
This work proposes a novel method for semi-supervised learning from partially labeled massive network-structured datasets, i.e., big data over networks. We model the underlying hypothesis, which relates data points to labels, as a graph signal, defined over some graph (network) structure intrinsic to the dataset. Follo…
Recent work by Jacot et al. (2018) has shown that training a neural network using gradient descent in parameter space is related to kernel gradient descent in function space with respect to the Neural Tangent Kernel (NTK). Lee et al. (2019) built on this result by establishing that the output of a neural network traine…
Zero-Copy Semantic Contagion: An In-Memory Streaming Architecture for Evolving Attention Graphscs.LG
Zero-Copy Architecture Detects Cross-Company Financial Signals Instantly.
problem Financial models miss cross-company disruptions due to static data.
method Heterogeneous Rust-Python streaming architecture that maps cross-company attention as a continuous-time graph.
result Zero-copy parsing and inference process delivers real-time cross-company signal detection.