arXiv research

A new robust regression method handles outliers in high-dimensional data.

We describe a general framework for measuring risks, where the risk measure takes values in an abstract cone. It is shown that this approach naturally includes the classical risk measures and set-valued risk measures and yields a natural definition of vector-valued risk measures. Several main constructions of risk meas…

New method solves sparse approximation problem using trimmed lasso and generalized soft-min penalties.

A new pseudo-metric uses data depth to compare probability distributions.

Theoretical limits of deep residual networks show consistent covariance structures.

Inverse depth scaling found in LLMs due to similar layers averaging error.

Residual networks with depthwise hyperparameter scaling transfer optimal hyperparameters across width and depth.

Investigates the relationship between ResNets and Neural ODEs, quantifying their closeness and providing training methods.

Residual networks' depth is mathematically equivalent to expanding an implicit ensemble size.

Unified learning-rate scale for CNNs and ResNets, avoiding depth imbalance.

One-shot neural architecture search limits depth search space and prunes networks for better performance and uncertainty.

Study infinite-depth limits of neural networks with fixed width.

Batch normalization makes deep residual networks train faster.

ResNets and DenseNets converge to NTK with depth and width, offering advantages for kernel regression.

Unified spectral framework for μP under joint width-depth scaling.

This paper analyzes Stochastic Depth regularization in ResNets.

Paper improves bike-sharing demand prediction by adapting to changing patterns.

The skip-connections used in residual networks have become a standard architecture choice in deep learning due to the increased training stability and generalization performance with this architecture, although there has been limited theoretical understanding for this improvement. In this work, we analyze overparameter…

Recent studies on automatic neural architectures search have demonstrated significant performance, competitive to or even better than hand-crafted neural architectures. However, most of the existing network architecture tend to use residual, parallel structures and concatenation block between shallow and deep features …

Study Transformer layers under cross-entropy training using mean field control.

Trimming helps in conformal prediction when it separates anomaly scores.

In this paper, we introduce transformations of deep rectifier networks, enabling the conversion of deep rectifier networks into shallow rectifier networks. We subsequently prove that any rectifier net of any depth can be represented by a maximum of a number of functions that can be realized by a shallow network with a …

We consider the problem of robustifying high-dimensional structured estimation. Robust techniques are key in real-world applications which often involve outliers and data corruption. We focus on trimmed versions of structurally regularized M-estimators in the high-dimensional setting, including the popular Least Trimme…

New method trims network data to resist adversarial contamination.

Nonconvex penalty methods for sparse modeling in linear regression have been a topic of fervent interest in recent years. Herein, we study a family of nonconvex penalty functions that we call the trimmed Lasso and that offers exact control over the desired level of sparsity of estimators. We analyze its structural prop…

We relate trimmed sums of twists in cylinders along a typical Teichmuller geodesic to the area Siegel-Veech constant.

Generalization bounds derived for neural ODEs and deep residual networks.

The study analyzes deep linear networks from random initialization, capturing dynamics and hyperparameter effects.

New core inflation measure predicts future headline inflation.

Gaussian Graphical Models (GGMs) are popular tools for studying network structures. However, many modern applications such as gene network discovery and social interactions analysis often involve high-dimensional noisy data with outliers or heavier tails than the Gaussian distribution. In this paper, we propose the Tri…

A method for estimating parameters from entangled single-sample distributions, robust to high-noise data.

Training deep recurrent neural network (RNN) architectures is complicated due to the increased network complexity. This disrupts the learning of higher order abstracts using deep RNN. In case of feed-forward networks training deep structures is simple and faster while learning long-term temporal information is not poss…

Sublinear functionals of random variables are known as sublinear expectations; they are convex homogeneous functionals on infinite-dimensional linear spaces. We extend this concept for set-valued functionals defined on measurable set-valued functions (which form a nonlinear space), equivalently, on random closed sets. …

Alpha-trimming prunes trees in random forests to improve predictive performance.

While training error of most deep neural networks degrades as the depth of the network increases, residual networks appear to be an exception. We show that the main reason for this is the Lyapunov stability of the gradient descent algorithm: for an arbitrarily chosen step size, the equilibria of the gradient descent ar…

MLP residual networks implement a selective coarse-graining procedure governed by the spectral structure of the input distribution.

New method prevents neural network breakdown by combining trimmed loss and variation regularization.

Our research proves neural collapse in deep ResNets and transformers is globally optimal.

We analyze the global convergence of gradient descent for deep linear residual networks by proposing a new initialization: zero-asymmetric (ZAS) initialization. It is motivated by avoiding stable manifolds of saddle points. We prove that under the ZAS initialization, for an arbitrary target matrix, gradient descent con…

Paper proposes robust gossip algorithms for mean and trimmed mean estimation.

SRFRN accelerates image super-resolution using shallow residual units.

TrIM improves gradient-based dimension reduction and regression.

New algorithm robustly estimates sparse models in high dimensions with corrupted data.

Estimates GLMs robustly against label corruptions.

We propose a robust elastic net (REN) model for high-dimensional sparse regression and give its performance guarantees (both the statistical error bound and the optimization bound). A simple idea of trimming the inner product is applied to the elastic net model. Specifically, we robustify the covariance matrix by trimm…

Deep residual networks implicitly converge to neural ODEs.

The Residual Network (ResNet), proposed in He et al. (2015), utilized shortcut connections to significantly reduce the difficulty of training, which resulted in great performance boosts in terms of both training and generalization error. It was empirically observed in He et al. (2015) that stacking more layers of resid…

This paper considers the problem of removing costly features from a Bayesian network classifier. We want the classifier to be robust to these changes, and maintain its classification behavior. To this end, we propose a closeness metric between Bayesian classifiers, called the expected classification agreement (ECA). Ou…