This paper focuses on a comparative evaluation of the most common and modern methods for text classification, including the recent deep learning strategies and ensemble methods. The study is motivated by a challenging real data problem, characterized by high-dimensional and extremely sparse data, deriving from incoming…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “shallow methods”
Sharp lower bounds on shallow neural networks' approximation rates are derived.
problem The efficiency of shallow neural networks in approximating functions.
method Lower bounding the -metric entropy and Kolmogorov -widths of the convex hull of neural network basis functions.
result Sharp lower bounds on the approximation rates for shallow neural networks are provided.
In this work, we present a comparison of a shallow and a deep learning architecture for the automated segmentation of white matter lesions in MR images of multiple sclerosis patients. In particular, we train and test both methods on early stage disease patients, to verify their performance in challenging conditions, mo…
Shallow neural networks can represent polynomials efficiently.
problem Representing polynomials using shallow neural networks.
method Using shallow neural networks of width to represent -variate polynomials of degree .
result Derives minimax optimal convergence rate for shallow networks to unknown univariate regression functions.
Deep ReLU networks approximate as well as shallow ones in kernel regimes.
problem Understanding the limitations of kernel methods for deep ReLU networks.
method Characterizing eigenvalue decays of kernels derived from deep ReLU networks.
result Deep ReLU networks and shallow two-layer networks have equivalent approximation properties in kernel regimes.
Proposes a learning rate method for shallow nets based on gradient Lipschitz constant.
problem Finding optimal learning rates for shallow neural networks.
method Associates learning rate with gradient Lipschitz constant and proposes a search algorithm.
result The proposed method significantly outperforms existing tuning methods.
New method for efficient proximal mapping of 1-path-norm in shallow networks.
problem Efficiently handling the 1-path-norm of shallow neural networks.
method Closed-form proximal operator for efficient computation and upper bound on Lipschitz constant.
result Proximal mapping allows robust training against adversarial perturbations.
This review unifies deep and shallow anomaly detection methods.
problem Improving anomaly detection on complex datasets.
method Systematic comparison of deep and shallow approaches.
result Identifies common principles and assumptions in anomaly detection.
Application of Machine Learning and Convex Limiting to Subgrid Flux Modeling in the Shallow-Water Equationsphysics.comp-ph
Combines machine learning and convex limiting for accurate subgrid flux modeling in shallow-water equations.
problem Accurate subgrid flux modeling in shallow-water equations.
method Machine learning and flux limiting for property-preserving subgrid scale modeling.
result The proposed method produces meaningful closures even in untrained scenarios.
Wide and shallow networks approximate convex functions well.
problem Understanding why wide and shallow neural networks perform well.
method Analyzing the epigraph of the input-output map of shallow and wide neural networks.
result The epigraph of the input-output map approximates a convex function.
Optimal rates for shallow ReLU networks in nonparametric regression.
problem Approximating smooth and non-smooth functions with shallow ReLU networks.
method Analysis of shallow ReLU neural networks, using variation norms and deep learning theory.
result Optimal approximation rates for shallow ReLU networks in nonparametric regression.
Convolutional Neural Networks for Text Categorization: Shallow Word-level vs. Deep Character-levelcs.CL
This paper reports the performances of shallow word-level convolutional neural networks (CNN), our earlier work (2015), on the eight datasets with relatively large training data that were used for testing the very deep character-level CNN in Conneau et al. (2016). Our findings are as follows. The shallow word-level CNN…
Deep ReLU networks can be simplified to a three-layer model.
problem Understanding the behavior of deep neural networks.
method Constructive proof and algorithm to transform deep networks into shallow ones.
result Deep ReLU networks can be represented by a simpler three-layer structure.
Doppler Invariant Demodulation for Shallow Water Acoustic Communications Using Deep Belief Networkseess.SP
Shallow water environments create a challenging channel for communications. In this paper, we focus on the challenges posed by the frequency-selective signal distortion called the Doppler effect. We explore the design and performance of machine learning (ML) based demodulation methods --- (1) Deep Belief Network-feed f…
In this paper, we address the issue of how to enhance the generalization performance of convolutional neural networks (CNN) in the early learning stage for image classification. This is motivated by real-time applications that require the generalization performance of CNN to be satisfactory within limited training time…
Deep neural networks can learn smooth functions without parameters.
problem Learning smooth functions from shallow ReLU neural networks.
method Using over-parameterized shallow ReLU neural networks with norm constraints.
result Least squares estimators based on shallow neural networks are minimax optimal.
Gradient descent trains shallow neural networks to approximate functions in 1D.
problem Approximating functions in 1D with shallow neural networks trained by gradient descent.
method Gradient descent optimization of non-convex weight space for finite width networks in 1D.
result Gradient descent can approximate functions in 1D with a minimal number of weights, balancing practical performance and theoretical capabilities.
A shallow Bi-LSTM model forecasts Bitcoin prices using engineered features.
problem Lack of high-quality historical Bitcoin data for deep learning models.
method Proposed a shallow Bi-LSTM model with feature engineering on Bitcoin data.
result A shallow deep neural network outperforms other models in Bitcoin price forecasting.
Neural networks minimize error with shallow ReLU models for function estimation.
problem Estimating unknown functions from noisy data.
method Minimizing squared errors plus weight decay regularization.
result Neural network estimators are minimax optimal up to logarithmic factors.
There is some theoretical evidence that deep neural networks with multiple hidden layers have a potential for more efficient representation of multidimensional mappings than shallow networks with a single hidden layer. The question is whether it is possible to exploit this theoretical advantage for finding such represe…
A new method reduces computational cost for gene expression inference in large microarray data sets.
problem Efficiently predicting gene expression in large datasets with limited resources.
method Adaptive Lipschitz constant inspired learning rate, random sub-sampling, and A-ReLU activation function.
result Remarkable improvement in saving computational cost while maintaining prediction accuracy.
CHOOSE enhances shallow Transformers for wireless symbol detection.
problem Improving wireless symbol detection with shallow Transformers.
method Introducing autoregressive latent reasoning steps within hidden space.
result Lightweight Transformers achieve comparable performance to deep models.
New method trains shallow neural networks with subquadratic width scaling.
problem Training shallow neural networks with optimal width scaling.
method Polyak-Lojasiewicz condition, smoothness, standard data assumptions, random matrix theory.
result Subquadratic scaling on network width with standard initialization strategies.
Extreme multi-label classification (XMC) refers to supervised multi-label learning involving hundreds of thousand or even millions of labels. In this paper, we develop a suite of algorithms, called Bonsai, which generalizes the notion of label representation in XMC, and partitions the labels in the representation space…
New bounds for shallow neural networks with deterministic parameters.
problem Developing generalisation bounds for shallow neural networks.
method PAC-Bayesian theory applied to shallow neural networks with deterministic parameters.
result Empirical non-vacuous bounds for shallow neural networks trained with vanilla SGD.
The paper analyzes stability and generalization of shallow neural networks using gradient methods.
problem Understanding the generalization of overparameterized shallow neural networks.
method The paper uses gradient descent and stochastic gradient descent to study shallow neural networks, developing consistent excess risk bounds.
result The analysis improves on existing methods by providing a refined estimation of iterates and Hessian eigenvalues, leading to better excess risk bounds.
In this work, we propose an iterative reconstruction scheme (ALONE - Adaptive Learning Of NEtworks) for 2D radial cine MRI based on ground truth-free unsupervised learning of shallow convolutional neural networks. The network is trained to approximate patches of the current estimate of the solution during the reconstru…
New method identifies parameters of wider shallow neural networks with biases.
problem Identifying parameters of wide shallow neural networks with biases from finite samples.
method Two-step pipeline: direction of weights via second order information, signs via algebraic evaluations, biases via gradient descent.
result Constructive methods and theoretical guarantees of finite sample identification for wider shallow networks with biases.
Explicit high-order feature interactions efficiently capture essential structural knowledge about the data of interest and have been used for constructing generative models. We present a supervised discriminative High-Order Parametric Embedding (HOPE) approach to data visualization and compression. Compared to deep emb…
New bound on neural nets complexity for approximating functions.
problem Approximating continuous functions with shallow neural networks.
method Inspired by Stone-Weierstrass theorem, constructive proof.
result General upper bound on neuron count for accuracy.
Study shows shallow ReLU networks struggle with high-dimensional Lipschitz functions.
problem Expressing high-dimensional Lipschitz functions with shallow ReLU networks.
method Established lower bounds on shallow network complexity for polynomial approximation.
result Shallow ReLU networks suffer from the curse of dimensionality for Lipschitz functions.
Sharp Bounds on the Approximation Rates, Metric Entropy, and -widths of Shallow Neural Networksstat.ML
Sharp bounds on neural network approximation rates and widths.
problem Estimating approximation rates, metric entropy, and n-widths of shallow neural networks.
method Introducing smoothly parameterized dictionaries and providing upper and lower bounds.
result Sharp bounds on approximation rates, metric entropy, and n-widths for neural networks with various activation functions.
Proves existence of optimal shallow neural networks with ReLU activation.
problem Proving the existence of optimal shallow feedforward networks with ReLU activation.
method Proves existence of global minima in the loss landscape for continuous target functions using shallow feedforward neural networks with ReLU activation.
result Existence of global minima in the loss landscape for shallow feedforward networks with ReLU activation.
Gradient-trained shallow networks can generalize well but are vulnerable to small-radius adversarial attacks.
problem Adversarial robustness of gradient-trained shallow networks.
method Analysis of neuron alignment and polynomial ReLU activation.
result Gradient-trained shallow networks with polynomial ReLU activation are robust to small-radius adversarial attacks.
In this paper, we introduce transformations of deep rectifier networks, enabling the conversion of deep rectifier networks into shallow rectifier networks. We subsequently prove that any rectifier net of any depth can be represented by a maximum of a number of functions that can be realized by a shallow network with a …
A new algorithm for learning shallow neural networks with infinite width.
problem Learning shallow over-parameterized neural networks.
method Sinkhorn proximal algorithm approximating mean field learning dynamics.
result The algorithm performs gradient descent of the free energy associated with the risk functional.
Study shows limits on deep and shallow neural networks for approximating compact sets.
problem Understanding the limitations of deep and shallow neural networks in approximating compact sets.
method Proved Carl's type inequalities for approximation error, using Lipschitz widths.
result Lower bounds on approximation error for neural network outputs.
Paper studies shallow ReLU networks' approximation rates for Hölder functions.
problem Understanding shallow ReLU networks' efficiency in approximating Hölder functions.
method Analyzes rates of uniform approximation by ReLU shallow neural networks with hidden neurons.
result Shows ReLU shallow neural networks can uniformly approximate Hölder functions with rates close to optimal.
Deep networks without non-linearities are equivalent to shallow ones.
problem Training deep orthogonal linear networks with no non-linearity.
method Riemannian gradient descent and gradient descent on factorization.
result Training deep overparametrized networks is equivalent to shallow ones.
Deep Learning the Ising Model Near Criticalitycond-mat.dis-nn
It is well established that neural networks with deep architectures perform better than shallow networks for many tasks in machine learning. In statistical physics, while there has been recent interest in representing physical data with generative modelling, the focus has been on shallow neural networks. A natural ques…
Unified theorem for deep and shallow joint-equivariant machines.
problem Universal approximation of joint-equivariant machines.
method Constructive universal approximation theorem based on ridgelet transform.
result Unified approximation of deep and shallow networks.
New neural networks learn single-index models efficiently.
problem Learning low-dimensional structure in high-dimensional data.
method Shallow neural networks with frozen biases, studied via gradient flow.
result Generalization guarantees match near-optimal sample complexity.
On the Impact of Overparameterization on the Training of a Shallow Neural Network in High Dimensionsmath.OC
Study shows overparameterization helps shallow neural networks recover signals in high dimensions.
problem Signal recovery in shallow neural networks with overparameterization.
method Gradient flow on population risk, Gaussian distribution assumption, high-dimensional limit analysis.
result Minimal overparameterization is sufficient for strong recovery of signals.
Deep networks learn hierarchical functions more efficiently than shallow ones.
problem Understanding the advantage of deep neural networks over shallow models.
method Analytical study of learning dynamics and generalization performance of deep networks compared to shallow ones.
result Deep networks reduce effective dimensionality, enabling learning with fewer samples.
Shallow trees in ensemble models make models more interpretable and sometimes better.
problem Lack of transparency in high-performing tree ensemble models.
method Developed an interpretation algorithm to convert tree ensembles into functional ANOVA representations. Proposed strategies to enhance interpretability.
result Shallow trees in ensemble models can lead to better generalization performance and improved interpretability.
The work concerns the problem of reducing a pre-trained deep neuronal network to a smaller network, with just few layers, whilst retaining the network's functionality on a given task The proposed approach is motivated by the observation that the aim to deliver the highest accuracy possible in the broadest range of oper…
This paper considers the power of deep neural networks (deep nets for short) in realizing data features. Based on refined covering number estimates, we find that, to realize some complex data features, deep nets can improve the performances of shallow neural networks (shallow nets for short) without requiring additiona…
This paper analyzes shallow ReLU networks in L^p and Sobolev spaces, focusing on approximation and generalization.
problem Approximation and generalization of shallow ReLU networks in L^p and Sobolev spaces.
method Spherical harmonic analysis and embeddings into spectral Barron spaces for L^p spaces, path-norm control for Sobolev spaces.
result Minimax-optimal rates for nonparametric regression with shallow ReLU networks under path-norm control.