STARK improves denoising of low-depth spatial transcriptomics images.
problem Denoising spatial transcriptomics images at ultra-low sequencing depths.
method Adaptive regularization with kernel ridge regression and graph Laplacian.
result STARK optimizes denoising performance over competing methods.
Transformers can learn Markov processes with constant depth, surprising results.
problem Understanding how transformers learn context in Markov processes.
method Empirical study and theoretical analysis of attention-based transformers on Markov data.
result Transformers with constant depth can achieve low test loss on Markov sequences, matching empirical and theoretical findings.
We consider rotationally symmetric spaces with low regularity, which we regard as integral currents spaces or manifolds with Sobolev regularity and are assumed to have nonnegative scalar curvature. Relying on the flat distance and on Sobolev norms, we establish several nonlinear stability estimates about the ``distance…
CoT enhances transformer accuracy on serial tasks by enabling serial computation.
problem Improving accuracy of large language models on inherently serial problems.
method Integrating a chain of thought (CoT) into decoder-only transformers to enable serial computation.
result Constant-depth transformers with CoT can solve problems in AC^0, surpassing TC^0 without CoT.
Deep architecture such as hierarchical semi-Markov models is an important class of models for nested sequential data. Current exact inference schemes either cost cubic time in sequence length, or exponential time in model depth. These costs are prohibitive for large-scale problems with arbitrary length and depth. In th…
Transformers simulate finite-state automata with fewer layers.
problem How do shallow, non-recurrent Transformers simulate complex computations?
method Hierarchical reparameterization of recurrent dynamics to simulate automata.
result Polynomial-sized, O(logT)-depth solutions exist and are common. In the last decade, supervised deep learning approaches have been extensively employed in visual odometry (VO) applications, which is not feasible in environments where labelled data is not abundant. On the other hand, unsupervised deep learning approaches for localization and mapping in unknown environments from unlab…
The theory of tunnel number 1 knots detailed in our previous paper, The tree of knot tunnels, provides a non-negative integer invariant called the depth of the tunnel. We give various results related to the depth invariant. Noting that it equals the minimum number of Goda-Scharlemann-Thompson tunnel moves needed to con…
Large neural networks learn low-dimensional representations that balance complexity and regularity.
problem Understanding the tradeoff between low-dimensional representations and complexity in deep neural networks.
method Computed finite depth corrections to reveal a measure of regularity that bounds the pseudo-determinant of the Jacobian.
result Proved the conjectured bottleneck structure in learned features as network depth increases, showing almost all hidden representations are approximately low-dimensional and weight matrices have singular values close to 1.
The angular power spectrum characterizes neural network complexity.
problem Characterizing the complexity of deep neural networks.
method Using the angular power spectrum of the limiting field to characterize network complexity.
result Classified neural networks as low-disorder, sparse, or high-disorder.
Robust machine learning models improve DNA regulatory sequence prediction under various shifts.
problem Real-world applications of DNA regulatory sequence prediction involve shifts not captured by standard i.i.d. assumptions.
method Introduces a robustness framework combining simulation benchmarks and real data analysis.
result Models remain accurate and calibrated under mild shifts but show higher error and miscalibration under strong shifts.
New method improves reliability of depth estimation models.
problem Uncertainty quantification in large-scale vision models.
method Parameter-efficient Bayesian neural networks with PEFT methods.
result Combining PEFT methods with Bayesian inference enhances predictive performance.
Transformer networks approximate Hölder and Sobolev functions with fixed-depth networks.
problem Nonparametric regression with dependent observations.
method Established novel upper bounds for Transformer networks approximating Hölder and Sobolev functions under various β-mixing data assumptions. result Explicit convergence rates for nonparametric regression problems under β-mixing data assumptions. Multidimensional recurrent neural networks (MDRNNs) have shown a remarkable performance in the area of speech and handwriting recognition. The performance of an MDRNN is improved by further increasing its depth, and the difficulty of learning the deeper network is overcome by using Hessian-free (HF) optimization. Given…
In order-driven markets, limit-order book (LOB) resiliency is an important microscopic indicator of market quality when the order book is hit by a liquidity shock and plays an essential role in the design of optimal submission strategies of large orders. However, the evolutionary behavior of LOB resilience around liqui…
Hypersolvers enable fast continuous-depth models for practical applications.
problem Infinite-depth models like Neural ODEs are computationally infeasible for large problems.
method Introducing hypersolvers, neural networks that solve ODEs efficiently with theoretical guarantees.
result Hypersolvers achieve comparable inference time to traditional discrete networks, making continuous-depth models practical.
seMCD computes depth functions with statistical guarantees using sequential Monte Carlo.
problem Computing depth functions is computationally challenging, especially in high dimensions.
method Sequential Monte Carlo methodology with theoretical and empirical guarantees.
result The seMCD method provides accurate depth approximations with fewer samples than traditional methods.
A generic geodesic on a finite area, hyperbolic 2-orbifold exhibits an infinite sequence of penetrations into a neighborhood of a cone singularity, so that the sequence of depths of maximal penetration has a limiting distribution. The distribution function is the same for all such surfaces and is described by a fairly …
Enhances SSL methods with depth cues for better image understanding.
problem Lack of depth cues in 2D image pixel maps limits SSL performance.
method Integrates depth signals from a pretrained monocular RGB-to-depth model into contrastive learning frameworks.
result Improves SSL methods' robustness and generalization with depth signals.
Recently, GAIL framework and various variants have shown remarkable possibilities for solving practical MDP problems. However, detailed researches of low-level, and high-dimensional state input in this framework, such as image sequences, has not been conducted. Furthermore, the cost function learned in the traditional …
UDN adapts depth to data complexity, outperforming standard neural networks.
problem Adapting neural network depth to data complexity.
method Variational inference for infinitely deep neural networks with a novel algorithm.
result UDN outperforms standard neural networks and other infinite-depth approaches.
New method finds unique branched surfaces in 3-manifolds.
problem Finding unique branched surfaces in 3-manifolds.
method Proving endperiodic maps have periodic splitting sequences.
result Veering branched surfaces are unique up to equivalence.
Paper uses statistical depth to create DP estimators for regression.
problem Creating differentially private estimators in high dimensions.
method Uses halfspace and regression depth to analyze maximum influence and construct DP estimators.
result New DP estimators for location and regression show favorable performance.
This note explores the consequences of nonlinear price impact functions on price dynamics within the chartist-fundamentalist framework. Price impact functions may be nonlinear with respect to trading volume. As indicated by recent empirical studies, a given transaction may cause a large (small) price change if market d…
Let f:Sd−1×Sd−1→S be a function of the form f(x,x′)=g(⟨x,x′⟩) for g:[−1,1]→R. We give a simple proof that shows that poly-size depth two neural networks with (exponentially) bounded weights cannot approximate $f…
Network quantization is an effective solution to compress deep neural networks for practical usage. Existing network quantization methods cannot sufficiently exploit the depth information to generate low-bit compressed network. In this paper, we propose two novel network quantization approaches, single-level network qu…
This is the second of three papers that refine and extend portions of our earlier preprint, "The depth of a knot tunnel." Together, they rework the entire preprint. The theory of tunnel number 1 knots that we introduced in "The tree of knot tunnels" yields a parameterization in which each tunnel is described uniquely b…
Depth uncertainty networks don't improve with bias correction, contrary to expectations.
problem Improving performance in active learning with overparameterised models like NNs.
method Depth uncertainty networks, compared to underparameterised models, show no improvement in performance with bias correction.
result Depth uncertainty networks do not improve with bias correction, unlike underparameterised models.
Combining explicit and implicit regularization improves deep learning performance without needing depth.
problem Improving deep learning performance without increasing model complexity.
method Proposes an explicit penalty to mirror implicit regularization bias in adaptive gradient optimizers.
result Single-layer networks can achieve low-rank approximations with similar performance to deep linear networks.
LoRA enhances model adaptability without increasing parameters.
problem Theoretical understanding of LoRA's adaptability.
method Theoretical analysis of LoRA's expressive power.
result LoRA can adapt any model to a smaller target model with a specific rank.
Prediction suffix trees (PST) provide an effective tool for sequence modelling and prediction. Current prediction techniques for PSTs rely on exact matching between the suffix of the current sequence and the previously observed sequence. We present a provably correct algorithm for learning a PST with approximate suffix…
We present a new approach to modeling sequential data: the deep equilibrium model (DEQ). Motivated by an observation that the hidden layers of many existing deep sequence models converge towards some fixed point, we propose the DEQ approach that directly finds these equilibrium points via root-finding. Such a method is…
Study explores how dataset breadth and depth affect Siamese Neural Network performance.
problem Impact of dataset breadth and depth on Siamese Neural Network performance.
method Experiments with three keystroke datasets varying breadth and depth factors.
result Increasing dataset breadth improves model performance, while depth's impact varies by dataset type.
Deep imagination optimizes decision-making in large trees with limited resources.
problem Optimal planning in large decision trees with limited resources and time.
method Analytical solutions and numerical analysis of sampling capacity allocation.
result Optimal policy is to allocate few samples per level for deep exploration, favoring depth over breadth.
Generation of pseudorandom numbers from different probability distributions has been studied extensively in the Monte Carlo simulation literature. Two standard generation techniques are the acceptance-rejection and inverse transformation methods. An alternative approach to Monte Carlo simulation is the quasi-Monte Carl…
New deep learning method outperforms random training data.
problem Improving accuracy of deep learning algorithms in high dimensions.
method Training with low-discrepancy sequences instead of random data.
result Significantly outperforms standard deep learning algorithms.
Deep RNNs excel at capturing long-term dependencies in sequential data.
problem Lack of a formal measure for RNNs' long-term memory capacity.
method Introduced a measure called Start-End separation rank to quantify RNNs' ability to model long-term dependencies.
result Deep RNNs support Start-End separation ranks that are combinatorially higher than shallow ones.
Tensor networks are a powerful modeling framework developed for computational many-body physics, which have only recently been applied within machine learning. In this work we utilize a uniform matrix product state (u-MPS) model for probabilistic modeling of sequence data. We first show that u-MPS enable sequence-level…
New method learns low-dimensional representations of nonlinear time series without supervision.
problem Learning low-dimensional representations of nonlinear time series without supervision.
method Based on monotone variational inequality, the method learns representations by assuming sequences arise from a common domain.
result The method can learn the geometry for the entire domain and faithful representations for the dynamics of each individual sequence.
Deep networks can perfectly classify two low-dimensional manifolds on a sphere with large depth and width.
problem Binary classification of two low-dimensional submanifolds on a sphere.
method Analysis of a deep fully-connected neural network trained to separate two submanifolds of the unit sphere.
result Randomly-initialized gradient descent can perfectly classify the two manifolds with high probability when the network depth is large relative to certain geometric and statistical properties of the data.
A new autoencoder architecture captures multiscale data.
problem Multiscale spatio-temporal data representation.
method Integrates multigrid methods, convolutional autoencoders, and transfer learning.
result Adaptive, hierarchical architecture captures different scaled features dynamically.
WGANs improve probability distribution approximation with depth and width trade-offs.
problem Approximating complex probability distributions accurately.
method Wasserstein GANs with GroupSort discriminators, quantified generalization bound.
result High-capacity discriminators are crucial for WGANs' performance.
LMC improves sampling from complex distributions using quasi-random sequences.
problem Sampling from complex high-dimensional distributions with high accuracy.
method Using completely uniformly distributed (CUD) sequences in Langevin Monte Carlo (LMC) to generate Gaussian perturbations.
result LMC with low-discrepancy CUD sequences achieves smaller estimation error than standard LMC.
Seq2Tens uses tensors to efficiently represent sequences, improving performance on time series and video tasks.
problem Challenges in analyzing sequential data due to complex dependencies and non-commutativity.
method Uses tensor algebra to capture dependencies and low-rank tensor projections to manage computational complexity.
result State-of-the-art performance on multivariate time series classification and video generation benchmarks.
RAGIC predicts stock intervals with risk considerations, improving prediction accuracy and coverage.
problem Limited success in predicting stock market outcomes due to stochastic nature and risk oversight.
method RAGIC uses a GAN with a risk module and temporal module to generate risk-sensitive stock intervals.
result RAGIC achieves a consistent 95% coverage with narrow interval widths, balancing accuracy and risk.
We propose an analytically tractable class of models for the dynamics of a limit order book, described through a stochastic partial differential equation (SPDE) with multiplicative noise for the order book centered at the mid-price, along with stochastic dynamics for the mid-price which is consistent with the order flo…
We introduce a new family of deep neural network models. Instead of specifying a discrete sequence of hidden layers, we parameterize the derivative of the hidden state using a neural network. The output of the network is computed using a black-box differential equation solver. These continuous-depth models have constan…
BERT embeddings improve sequence quality metrics.
problem Measuring the quality of generated sequences against references.
method Employ contextual BERT embeddings for sequence-level reward.
result Contextual embeddings provide a more effective learning signal.