arXiv research

The performance of standard learning procedures has been observed to differ widely across groups. Recent studies usually attribute this loss discrepancy to an information deficiency for one group (e.g., one group has less data). In this work, we point to a more subtle source of loss discrepancy---feature noise. Our mai…

Study predicts SGD test loss for structured features.

This paper examines how different loss functions affect neural network features and performance.

NGP selects N features from P using neural networks in a greedy, iterative process.

Fast classification for sparse models, even with correlated features.

Study shows how information loss and operation loss are related in feature representations.

Study feature representations induced by dependence between variables.

Q-SHAP efficiently calculates feature contributions in boosting trees.

This paper proposes a Convolutional Neural Network (CNN) inspired by Multitask Learning (MTL) and based on speech features trained under the joint supervision of softmax loss and center loss, a powerful metric learning strategy, for the recognition of emotion in speech. Speech features such as Spectrograms and Mel-freq…

Feature selection is a technique to screen out less important features. Many existing supervised feature selection algorithms use redundancy and relevancy as the main criteria to select features. However, feature interaction, potentially a key characteristic in real-world problems, has not received much attention. As a…

Contrastive learning properties studied, including feature suppression and hierarchical learning.

Cross-entropy loss together with softmax is arguably one of the most common used supervision components in convolutional neural networks (CNNs). Despite its simplicity, popularity and excellent performance, the component does not explicitly encourage discriminative learning of features. In this paper, we propose a gene…

This paper presents a learning method for convolutional autoencoders (CAEs) for extracting features from images. CAEs can be obtained by utilizing convolutional neural networks to learn an approximation to the identity function in an unsupervised manner. The loss function based on the pixel loss (PL) that is the mean s…

This paper concerns a method of selecting a subset of features for a sequential logit model. Tanaka and Nakagawa (2014) proposed a mixed integer quadratic optimization formulation for solving the problem based on a quadratic approximation of the logistic loss function. However, since there is a significant gap between …

The paper explains spikes in training loss as catapults, improving feature learning and generalization.

We prove that, under low noise assumptions, the support vector machine with $N\ll m$ random features (RFSVM) can achieve the learning rate faster than $O(1/\sqrt{m})$ on a training set with $m$ samples when an optimized feature map is used. Our work extends the previous fast rate analysis of random features method from…

RFRBoost uses random features to boost deep residual neural networks, improving performance and computational efficiency.

Grokking occurs when neural networks transition from lazy to rich training dynamics, fitting initial features before generalizing.

t-SNE loses important features in data visualization.

This paper extends neural collapse to imbalanced data under cross-entropy loss.

A method to identify important features without solving the full problem.

Paper uses deep Ritz method for solving stationary Schrödinger equation, proving convergence and feature emergence.

This paper proposes a Residual Convolutional Neural Network (ResNet) based on speech features and trained under Focal Loss to recognize emotion in speech. Speech features such as Spectrogram and Mel-frequency Cepstral Coefficients (MFCCs) have shown the ability to characterize emotion better than just plain text. Furth…

Topology-enhanced loss improves 3D object reconstruction from 2D images.

We consider the problem of learning linear classifiers when both features and labels are binary. In addition, the features are noisy, i.e., they could be flipped with an unknown probability. In Sy-De attribute noise model, where all features could be noisy together with same probability, we show that $0$-$1$ loss ($l_{…

Paper proposes D3M to improve anti-spoofing detection by balancing loss function and using complementary features.

Previous work shows that adversarially robust generalization requires larger sample complexity, and the same dataset, e.g., CIFAR-10, which enables good standard accuracy may not suffice to train robust models. Since collecting new training data could be costly, we focus on better utilizing the given data by inducing t…

New tools reveal simple structure in complex hyperparameter loss surfaces near optima.

Over-parameterized models reduce Out-of-Distribution (OOD) generalization loss.

In this paper we study deep learning-based music source separation, and explore using an alternative loss to the standard spectrogram pixel-level L2 loss for model training. Our main contribution is in demonstrating that adding a high-level feature loss term, extracted from the spectrograms using a VGG net, can improve…

This paper proposes a novel generic one-class feature learning method based on intra-class splitting. In one-class classification, feature learning is challenging, because only samples of one class are available during training. Hence, state-of-the-art methods require reference multi-class datasets to pretrain feature …

Feature normalization prevents collapse in non-contrastive learning dynamics.

This work justifies neural collapse under MSE loss and analyzes the optimization landscape.

Deep nets trained with MSE loss exhibit Neural Collapse, collapsing features and classifiers to class means.

A new framework for deep matrix factorizations improves model consistency and flexibility.

The paper decomposes probabilistic scores into reliability, uncertainty, and information loss.

In this paper, we focus on the separability of classes with the cross-entropy loss function for classification problems by theoretically analyzing the intra-class distance and inter-class distance (i.e. the distance between any two points belonging to the same class and different classes, respectively) in the feature s…

Wide neural networks converge linearly to zero loss with feature learning.

Despite the remarkable performance of deep neural networks on various computer vision tasks, they are known to be susceptible to adversarial perturbations, which makes it challenging to deploy them in real-world safety-critical applications. In this paper, we conjecture that the leading cause of adversarial vulnerabili…

This paper extends neural collapse to class-imbalanced datasets using an unconstrained ReLU feature model.

DNNs with $L_{2}$ regularization reveal feature learning dynamics and sparsity.

Deep RL policies share adversarial features across different MDPs.

Toy model study shows resampling/reweighting can improve feature learning in imbalanced classification.

This work interprets SFA through variational inference, relaxing linearity constraints.

Current methods to interpret deep learning models by generating saliency maps generally rely on two key assumptions. First, they use first-order approximations of the loss function neglecting higher-order terms such as the loss curvatures. Second, they evaluate each feature's importance in isolation, ignoring their int…

Enhanced loss function boosts fraud detection in auto insurance claims.

We present $α$-loss, $α\in [1,\infty]$, a tunable loss function for binary classification that bridges log-loss ($α=1$) and $0$-$1$ loss ($α= \infty$). We prove that $α$-loss has an equivalent margin-based form and is classification-calibrated, two desirable properties for a good surrogate loss function for the ideal y…

Framework explains how dual deep networks learn features from unlabeled data.