CNN improves neutrino event reconstruction in IceCube DeepCore.
problem Difficulties in distinguishing muon neutrinos and reconstructing inelasticity at GeV scale energies.
method 2D Convolutional Neural Network exploiting time and depth translational symmetry.
result CNN model outperforms conventional methods for flavor identification and inelasticity reconstruction.
New model preserves symmetry in multivariate time series, improving performance.
problem Implicit ordering in MTS models violates inherent exchangeability.
method Permutation-equivariant 2D state space model with canonical architecture.
result Eliminates sequential dependency chains and simplifies stability analysis.
Wide neural networks learn features under μP, identifying weights and decomposing support.
problem Feature learning in wide neural networks under μP. method Proving mean-field limit, characterizing identifiability, sparse-dictionary decomposition, and feature-learning-error decomposition.
result The triple (w∗,Dorb∗,S∗) identifies the natural learning cell of the architecture-data pair (σ,ρ). Deep neural networks favor symmetric structures, enabling multilevel symmetries.
problem Understanding and optimizing deep neural networks.
method Formulating DNN training as convex Lasso problems with geometric algebra.
result Deep networks inherently favor symmetric structures, enabling multilevel symmetries.
Our research proves neural collapse in deep ResNets and transformers is globally optimal.
problem Understanding neural collapse in deep learning models.
method Analysis of deep regularized transformers and ResNets trained with cross entropy or mean squared error loss.
result Global optima of deep regularized transformers and ResNets are approximately collapsed, becoming more prominent as depth increases.
Theory explains deep nonlinear networks' plateaus and transitions.
problem Understanding long plateaus and feature acquisition transitions in deep nonlinear networks.
method Derived an exact identity for Frobenius norms, classified activation functions, and reduced matrix flow to a scalar ODE.
result Escape time law τ⋆=Θ(ε−(r−2)) for deep nonlinear networks, where r is the number of bottleneck layers. New findings on hidden symmetries in ReLU networks.
problem Understanding the redundancy and symmetries in ReLU network parameter space.
method Analyzing parameter settings and function classes for various network architectures.
result For certain network architectures, there are no hidden symmetries.
The paper studies randomized approximations of Tukey's depth for log-concave isotropic data.
problem The challenge of approximating Tukey's depth in high dimensions.
method The study examines randomized algorithms for approximating Tukey's depth for log-concave isotropic data.
result Randomized algorithms correctly approximate maximal depth and close to zero depths but not intermediate depths.
New algorithm learns random neural networks efficiently.
problem Learning random constant-depth neural networks efficiently.
method Presented a PTAS (Polynomial-Time Approximation Scheme) for learning random Xavier networks of fixed depth.
result For any fixed ε and depth i, there is a poly-time algorithm that learns random Xavier networks up to an additive error of ε.
Entangled embedded periodic nets and crystal frameworks are defined, along with their dimension type, homogeneity type, adjacency depth and periodic isotopy type. We obtain periodic isotopy classifications for various families of embedded nets with small quotient graphs. We enumerate the 25 periodic isotopy classes of …
Noise balance theory explains SGD's behavior in neural networks.
problem Understanding SGD's navigation in neural network loss landscapes.
method Analyzes minibatch noise and loss function symmetries.
result Derives the stationary distribution of SGD for deep networks.
This paper explores Bayesian Neural Network posteriors, uncovering symmetries and their impact.
problem Understanding the complex posterior distribution of deep Bayesian Neural Networks.
method Investigates optimal approaches for approximating posteriors, analyzes modes, and explores visualizations.
result Uncovered weight-space symmetries and their impact on the posterior, particularly scaling symmetries.
Let f:Sd−1×Sd−1→S be a function of the form f(x,x′)=g(⟨x,x′⟩) for g:[−1,1]→R. We give a simple proof that shows that poly-size depth two neural networks with (exponentially) bounded weights cannot approximate $f…
The global symmetry algebras of partially-massless (PM) higher-spin (HS) fields in (A)dSd+1 are studied. The algebras involving PM generators up to depth 2(ℓ−1) are defined as the maximal symmetries of free conformal scalar field with 2ℓ order wave equation in d dimensions. We review the constructi…
Deep architecture such as hierarchical semi-Markov models is an important class of models for nested sequential data. Current exact inference schemes either cost cubic time in sequence length, or exponential time in model depth. These costs are prohibitive for large-scale problems with arbitrary length and depth. In th…
Depth is a key component of Deep Neural Networks (DNNs), however, designing depth is heuristic and requires many human efforts. We propose AutoGrow to automate depth discovery in DNNs: starting from a shallow seed architecture, AutoGrow grows new layers if the growth improves the accuracy; otherwise, stops growing and …
One-shot neural architecture search limits depth search space and prunes networks for better performance and uncertainty.
problem Finding optimal depth in residual networks for efficient training and inference.
method Formulated a variational objective to approximate the depth distribution and pruned networks based on this distribution.
result Pruned networks achieve competitive accuracy with unpruned networks and better uncertainty calibration.
We introduce SARR for symmetric object pose estimation, improving CNN performance.
problem Ambiguities in symmetric object orientations hinder deep learning pose estimation.
method Numeric rotation representation using symmetry-derived trigonometric identities.
result SARR enables standard CNNs to achieve state-of-the-art performance.
Data depth aids in identifying anomalies in multivariate data.
problem Detecting abnormal observations in multivariate datasets.
method Using data depth to assign abnormality labels to observations with lower depth values.
result Data depth effectively identifies anomalies in multivariate settings.
We study the complexity of functions computable by deep feedforward neural networks with piecewise linear activations in terms of the symmetries and the number of linear regions that they have. Deep networks are able to sequentially map portions of each layer's input-space to the same output. In this way, deep models c…
We describe a method to infer dense depth from camera motion and sparse depth as estimated using a visual-inertial odometry system. Unlike other scenarios using point clouds from lidar or structured light sensors, we have few hundreds to few thousand points, insufficient to inform the topology of the scene. Our method …
We show that the standard stochastic gradient decent (SGD) algorithm is guaranteed to learn, in polynomial time, a function that is competitive with the best function in the conjugate kernel space of the network, as defined in Daniely, Frostig and Singer. The result holds for log-depth networks from a rich family of ar…
Study on feature learning dynamics in infinite-depth neural networks, focusing on ResNets.
problem Understanding how features evolve during training in deep neural networks, especially in the large-depth limit.
method Conditional Gaussian representations and SDE system with decoupled backward weights.
result Depth-induced suppression of forward-backward coupling in infinite-depth networks, leading to a decoupled forward-backward SDE system.
Study reveals how depth of reasoning affects generalization in models.
problem Understanding scaling behavior of generalization with CoT depth.
method Theoretical model of CoT in linear regression using random matrix theory.
result Sharp phase transition between exponential and polynomial improvement, saturation, and overthinking.
Untrained neural networks can unfairly assign predictions to the same class.
problem Biasing effects in neural networks during initial training phases.
method Theoretical analysis of deep neural networks, focusing on Initial Guessing Bias (IGB).
result Model structure and preprocessing methods influence IGB.
Hypersolvers enable fast continuous-depth models for practical applications.
problem Infinite-depth models like Neural ODEs are computationally infeasible for large problems.
method Introducing hypersolvers, neural networks that solve ODEs efficiently with theoretical guarantees.
result Hypersolvers achieve comparable inference time to traditional discrete networks, making continuous-depth models practical.
A new optimizer DDC improves deep learning models by respecting symmetries.
problem Deep networks' loss is invariant to continuous symmetries, leading to optimization issues.
method DDC builds a Dead-Direction Conditioner that lifts a base optimizer into a G-equivariant one, preserving the quotient geometry.
result DDCAdam and DDCMuon outperform standard optimizers in various tasks, improving validation-train loss gaps and learning dynamics.
Bayesian linear networks reveal optimal depth and width trade-offs.
problem Understanding how depth, width, and dataset size affect model quality in linear networks.
method Zero noise Bayesian inference with Gaussian weight priors and mean squared error.
result Optimal predictions at infinite depth and maximized Bayesian model evidence at infinite depth.
With the ubiquity of sensors in the IoT era, statistical observations are becoming increasingly available in the form of massive (multivariate) time-series. Formulated as unsupervised anomaly detection tasks, an abundance of applications like aviation safety management, the health monitoring of complex infrastructures …
New methods improve prediction performance and reduce computation time in boosting and random forest models.
problem Improving prediction performance and reducing computation time in boosting and random forest models.
method Random tree depth injection approach for Boosting and Random Forests.
result The new methods can improve prediction performance and reduce computation time by up to 40%.
TRS-ODENs learn dynamics with time-reversal symmetry for more efficient learning.
problem Learning dynamics with time-reversal symmetry for more efficient learning.
method Proposed a loss function and a new framework (TRS-ODENs) to learn dynamics efficiently.
result TRS-ODENs can learn dynamics from noisy and complex trajectories efficiently.
Polynomial time algorithm learns depth-2 neural networks with ReLU activations.
problem Learning depth-2 neural networks with non-zero bias terms and general ReLU activations.
method Robust tensor decomposition of Hermite expansions.
result Polynomial time and sample efficient learning of depth-2 networks with ReLU activations.
The purpose of the present article is to study and characterize sev- eral types of symmetries of generalized Robertson-Walker space-times. Con- formal vector fields, curvature and Ricci collineations are studied. Many im- plications for existence of these symmetries on generalied Robertson-Walker spacetimes are obtaine…
New steady gradient Ricci solitons found with specific symmetry.
problem Finding new steady gradient Ricci solitons with positive curvature.
method Utilized a procedure by Lai to construct examples with O(p)imesO(q) symmetry. result Found new examples of steady gradient Ricci solitons with O(p)imesO(q) symmetry in dimensions p+q. Continuous-depth Evoformer reduces protein folding prediction time and resource usage.
problem Efficient protein structure prediction with reduced computational costs.
method Continuous-depth formulation of Evoformer using Neural Ordinary Differential Equations (Neural ODEs).
result The continuous-time Evoformer achieves constant memory cost and improved efficiency.
A vector field is called a Beltrami vector field, if B×(∇×B)=0. In this paper we construct two unique Beltrami vector fields I and Y, such that ∇×I=I, ∇×Y=Y, and such that both have an orientation-preserving …
DUNs improve active learning by dynamically adjusting model complexity.
problem Active learning datasets grow, leading to model bias or overfitting.
method Depth Uncertainty Networks (DUNs) infer model depth for flexibility and complexity.
result DUNs outperform other BNN variants on active learning tasks, reducing overfitting.
Algorithm efficiently learns deep ReLU networks with polynomial runtime in depth and parameters.
problem Learning deep ReLU networks with polynomial runtime.
method Algorithm using filtered PCA and lattice polynomial analysis.
result First nontrivial results for networks of depth more than two with polynomial runtime.
GoTube verifies neural networks over time, scaling to large horizons.
problem Verifying the robustness of time-continuous neural networks.
method Solves Go problems to construct a conservative execution set.
result Substantially outperforms existing tools in size, speed, and scalability.
Recurrent Neural Networks (RNNs) play a major role in the field of sequential learning, and have outperformed traditional algorithms on many benchmarks. Training deep RNNs still remains a challenge, and most of the state-of-the-art models are structured with a transition depth of 2-4 layers. Recurrent Highway Networks …
Deep RNNs excel at capturing long-term dependencies in sequential data.
problem Lack of a formal measure for RNNs' long-term memory capacity.
method Introduced a measure called Start-End separation rank to quantify RNNs' ability to model long-term dependencies.
result Deep RNNs support Start-End separation ranks that are combinatorially higher than shallow ones.
Symmetry, a central concept in understanding the laws of nature, has been used for centuries in physics, mathematics, and chemistry, to help make mathematical models tractable. Yet, despite its power, symmetry has not been used extensively in machine learning, until rather recently. In this article we show a general wa…
The symmetries of complex molecular structures can be modeled by the {\em topological symmetry group} of the underlying embedded graph. It is therefore important to understand which topological symmetry groups can be realized by particular abstract graphs. This question has been answered for complete graphs; it is natu…
In this paper we introduce a novel method of gradient normalization and decay with respect to depth. Our method leverages the simple concept of normalizing all gradients in a deep neural network, and then decaying said gradients with respect to their depth in the network. Our proposed normalization and decay techniques…
Method combines LD and Fermat Distance for neural network uncertainty.
problem Measuring uncertainty in neural network predictions.
method Statistical Depth (LD) combined with Fermat Distance.
result Effective uncertainty estimation without impacting original model performance.
New insights into how depth and width affect in-context learning in deep models.
problem Understanding how various resources impact in-context learning in deep models.
method Analyzed linear regression in a deep linear self-attention model, varying resources like depth, width, context length, and training steps.
result Increasing depth improves in-context learning even at infinite context length, contrary to previous findings.
The paper introduces new metrics on Finsler manifolds and characterizes associated vector fields.
problem Characterizing vector fields on Finsler manifolds with new metrics.
method Introducing F-natural metrics and characterizing conformal, homothetic, and Killing vector fields. result Characterization of vector fields on slit tangent bundles of Finsler manifolds.
Deep convolutional neural networks (CNNs) used in practice employ potentially hundreds of layers and 10,000s of nodes. Such network sizes entail significant computational complexity due to the large number of convolutions that need to be carried out; in addition, a large number of parameters needs to be learned and…