We study adversarial robustness of neural networks from a margin maximization perspective, where margins are defined as the distances from inputs to a classifier's decision boundary. Our study shows that maximizing margins can be achieved by minimizing the adversarial loss on the decision boundary at the "shortest succ…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,657 papers · 148 categories
Trend · papers per month
48 results for “Margin maximization”
Support vector regression (SVR) is one of the most popular machine learning algorithms aiming to generate the optimal regression curve through maximizing the minimal margin of selected training samples, i.e., support vectors. Recent researchers reveal that maximizing the margin distribution of whole training dataset ra…
The ultimate goal of a supervised learning algorithm is to produce models constructed on the training data that can generalize well to new examples. In classification, functional margin maximization -- correctly classifying as many training examples as possible with maximal confidence --has been known to construct mode…
Proposes a method to optimize neural network initialization using marginal likelihood maximization.
problem Optimizing hyperparameters for neural network initialization.
method Leverages the connection between neural networks and Gaussian processes to infer optimal hyperparameters.
result Marginal likelihood maximization provides near-optimal prediction performance on MNIST classification tasks.
In critical decision-making scenarios, optimizing accuracy can lead to a biased classifier, hence past work recommends enforcing group-based fairness metrics in addition to maximizing accuracy. However, doing so exposes the classifier to another kind of bias called infra-marginality. This refers to individual-level bia…
Paper analyzes GMM for separable data with various parameter structures.
problem Classifying separable data with logistic models and their generalizations.
method Introduces and analyzes Generalized Margin Maximizer (GMM) for logistic models with specific parameter structures.
result GMM outperforms max-margin classifiers in various parameter settings and structures.
A fast method for training linear classifiers maximizes margins.
problem Training linear classifiers with maximum margins.
method Momentum-based gradient method derived from convex dual with Nesterov acceleration.
result Exponentially faster convergence rate compared to standard methods.
We study the parameter estimation problem in mixture models with observational nonidentifiability: the full model (also containing hidden variables) is identifiable, but the marginal (observed) model is not. Hence global maxima of the marginal likelihood are (infinitely) degenerate and predictions of the marginal likel…
The paper proposes effective margin regularization to improve adversarial robustness in deep neural networks.
problem Adversarial vulnerability of deep neural networks (DNNs).
method Regularization of effective weight norm during training to maximize effective margins.
result Effective margin regularization (EMR) boosts adversarial robustness in both standard and adversarial training.
The paper studies marginally trapped submanifolds in Lorentzian manifolds under null energy condition.
problem Understanding marginally trapped submanifolds in Lorentzian manifolds.
method Analyzes properties of marginally trapped submanifolds in a Lorentzian manifold satisfying the null energy condition.
result Marginally trapped submanifolds have locally volume-maximizing properties in certain null hypersurfaces.
New insights into how linear classifiers and leaky ReLU networks can overfit without harming generalization.
problem Understanding conditions for benign overfitting in linear classifiers and leaky ReLU networks.
method Utilizing Karush--Kuhn--Tucker (KKT) conditions for margin maximization.
result Satisfaction of KKT conditions leads to benign overfitting in linear classifiers and leaky ReLU networks.
The marginal maximum a posteriori probability (MAP) estimation problem, which calculates the mode of the marginal posterior distribution of a subset of variables with the remaining variables marginalized, is an important inference problem in many models, such as those with hidden variables or uncertain parameters. Unfo…
We define a generalized likelihood function based on uncertainty measures and show that maximizing such a likelihood function for different measures induces different types of classifiers. In the probabilistic framework, we obtain classifiers that optimize the cross-entropy function. In the possibilistic framework, we …
New integrable systems for marginally trapped surfaces in 4D Lorentz-Minkowski space.
problem Constructing new representations for marginally trapped surfaces in .
method Developed new Weierstrass-type representations to solve a linear PDE.
result Explicit examples of marginally trapped surfaces with non-vanishing mean curvature.
In the absence of prior knowledge, ordinal embedding methods obtain new representation for items in a low-dimensional Euclidean space via a set of quadruple-wise comparisons. These ordinal comparisons often come from human annotators, and sufficient comparisons induce the success of classical approaches. However, colle…
Boosting algorithms produce a classifier by iteratively combining base hypotheses. It has been observed experimentally that the generalization error keeps improving even after achieving zero training error. One popular explanation attributes this to improvements in margins. A common goal in a long line of research, is …
Study shows momentum-based optimizers like Muon and MomentumGD bias towards KKT points in smooth homogeneous models.
problem Understanding the implicit bias of momentum-based optimizers on smooth homogeneous models.
method Analysis of Muon, MomentumGD, Signum, and Adam optimizers under decaying learning rate schedules.
result Momentum-based optimizers approximate steepest descent trajectories and bias towards KKT points of margin maximization problems.
New DAM method improves AUC scores in medical image classification.
problem Maximizing AUC in large-scale medical image classification.
method Proposes AUC margin loss for robust optimization, conducts extensive empirical studies.
result Improves performance on four medical image classification tasks, achieving 1st place on Stanford CheXpert.
A popular heuristic for improved performance in Generative adversarial networks (GANs) is to use some form of gradient penalty on the discriminator. This gradient penalty was originally motivated by a Wasserstein distance formulation. However, the use of gradient penalty in other GAN formulations is not well motivated.…
New method estimates Gaussian copulas with missing data using EM algorithm.
problem Estimating Gaussian copulas with missing data and prior assumptions.
method Rigorous application of the Expectation Maximization (EM) algorithm for marginal distributions and dependence structure.
result Joint distribution learned is closer to the underlying distribution.
The paper improves SVM margin-based generalization bounds.
problem Improving generalization bounds for SVMs.
method Revisiting and improving classic generalization bounds in terms of margins, complementing with a nearly matching lower bound.
result Almost settles the generalization performance of SVMs in terms of margins.
This work improves adversarial robustness by boosting model ensembles with margin maximization.
problem Single models are insufficient for defending against adversarial attacks.
method Margin-boosting approach to learn ensembles with maximum margin.
result Our algorithm outperforms existing ensembling techniques and large models trained end-to-end.
Improves EM algorithm for better local optima in mixture models.
problem EM algorithm's sensitivity to initialization and bad local optima.
method Big Learning principle applied to upgrade EM algorithm.
result BigLearn-EM delivers optimal solution with high probability.
Risk bounds for Classification and Regression Trees (CART, Breiman et. al. 1984) classifiers are obtained under a margin condition in the binary supervised classification framework. These risk bounds are obtained conditionally on the construction of the maximal deep binary tree and permit to prove that the linear penal…
Most real world phenomena such as sunlight distribution under a forest canopy, minerals concentration, stock valuation, exhibit nonstationary dynamics i.e. phenomenon variation changes depending on the locality. Nonstationary dynamics pose both theoretical and practical challenges to statistical machine learning algori…
We propose a practical Bayesian optimization method using Gaussian process regression, of which the marginal likelihood is maximized where the number of model selection steps is guided by a pre-defined threshold. Since Bayesian optimization consumes a large portion of its execution time in finding the optimal free para…
In this paper, we study the implicit regularization of the gradient descent algorithm in homogeneous neural networks, including fully-connected and convolutional neural networks with ReLU or LeakyReLU activations. In particular, we study the gradient descent or gradient flow (i.e., gradient descent with infinitesimal s…
Study shows how steepest descent algorithms' geometric margin increases during training.
problem Understanding implicit bias in steepest descent algorithms for neural networks.
method Analysis of steepest descent algorithms with infinitesimal learning rates in homogeneous neural networks.
result Limit points of training trajectories correspond to KKT points of margin-maximization problems.
New loss function improves convergence rate for neural networks.
problem Improving convergence speed of neural networks for classification tasks.
method Proposes a modified hinge loss function with gradients to critical points.
result Margin converges to max-margin at rate, faster than exponential losses.
The problem of maximizing (or minimizing) the agreement between clusterings, subject to given marginals, can be formally posed under a common framework for several agreement measures. Until now, it was possible to find its solution only through numerical algorithms. Here, an explicit solution is shown for the case wher…
Entropy regularization is used to get improved optimization performance in reinforcement learning tasks. A common form of regularization is to maximize policy entropy to avoid premature convergence and lead to more stochastic policies for exploration through action space. However, this does not ensure exploration in th…
Paper corrects Max-Margin loss for multi-label tasks.
problem Max-Margin loss inconsistency in multi-label classification.
method Introduced Restricted-Max-Margin loss.
result Consistent loss for multi-label tasks under milder conditions.
GD iterates for non-homogeneous deep nets increase margin and converge in direction.
problem Understanding implicit bias in non-homogeneous deep networks.
method Characterization of GD iterates' properties starting from small empirical risk.
result GD iterates converge in direction despite diverging norms, satisfying KKT conditions.
Proposes a novel SVM model for binary classification with different misclassification costs.
problem Real-world classification problems with varying misclassification costs.
method Incorporates performance constraints in SVM formulation to seek a hyperplane with maximal margin and misclassification rates below given thresholds.
result The proposed model gives users control over misclassification rates in one class at the expense of the other.
New findings on the max margin problem in neural networks.
problem Understanding the max margin problem in neural networks.
method Analyzing gradient flow and max margin problem in linear and ReLU networks.
result The KKT point is not always an optimum of the max margin problem.
In this work, we study a new approach to optimizing the margin distribution realized by binary classifiers. The classical approach to this problem is simply maximization of the expected margin, while more recent proposals consider simultaneous variance control and proxy objectives based on robust location estimates, in…
Unified framework for accelerated Perceptron and related problems.
problem Finding optimal linear threshold functions for classification.
method Modern acceleration techniques, specifically optimistic online learning.
result Improved convergence rates for various Perceptron-related problems.
Training deep generative models with maximum likelihood remains a challenge. The typical workaround is to use variational inference (VI) and maximize a lower bound to the log marginal likelihood of the data. Variational auto-encoders (VAEs) adopt this approach. They further amortize the cost of inference by using a rec…
With an eye toward understanding complexity control in deep learning, we study how infinitesimal regularization or gradient descent optimization lead to margin maximizing solutions in both homogeneous and non-homogeneous models, extending previous work that focused on infinitesimal regularization only in homogeneous mo…
New insights into deep learning: reducing training data significantly improves performance.
problem Understanding and improving generalization in deep learning models.
method Analyzing the distribution of classification margins and dynamically reducing the training set.
result The area under the curve of the margin distribution is a good measure of generalization.
We show that the maximal future development of asymptotically flat spherically symmetric black hole initial data for a self-gravitating nonlinear scalar field, also called a Higgs field, contains a connected, achronal marginally trapped tube which is asymptotic to the event horizon of the black hole, provided the initi…
We present novel understandings of the Gamma-Poisson (GaP) model, a probabilistic matrix factorization model for count data. We show that GaP can be rewritten free of the score/activation matrix. This gives us new insights about the estimation of the topic/dictionary matrix by maximum marginal likelihood estimation. In…
Neural models improve GLMMs for complex data.
problem Nonlinear relationships in grouped data.
method Replaced linear function with neural networks.
result Improved performance on synthetic and real-world data.
Deep networks converge in direction, with implications for predictions and margins.
problem Understanding convergence and alignment in deep learning networks.
method Developed a theory of unbounded nonsmooth Kurdyka-Łojasiewicz inequalities for functions definable in an o-minimal structure.
result Network weights, predictions, training errors, and margin distribution converge in direction and align with gradient flow.
This paper solves robust utility maximization with unknown claim dependencies.
problem Investor optimizes utility in the presence of an intractable contingent claim.
method Quantile optimization approach, transforming dynamic problem into static concave optimization.
result Optimal payoffs depend on ambiguity attitude, market conditions, and claim characteristics.
Adversarial robustness has become an important research topic given empirical demonstrations on the lack of robustness of deep neural networks. Unfortunately, recent theoretical results suggest that adversarial training induces a strict tradeoff between classification accuracy and adversarial robustness. In this paper,…
We obtain bounds on the distribution of the maximum of a martingale with fixed marginals at finitely many intermediate times. The bounds are sharp and attained by a solution to -marginal Skorokhod embedding problem in Obłój and Spoida [An iterated Azéma-Yor type embedding for finitely many marginals (2013) Preprint]…
Max-margin classifiers' behavior is studied in high dimensions with non-Gaussian features.
problem Understanding the role of featurization maps and high-dimensional misclassification error.
method High-dimensional asymptotics, Gaussian model, support vector representation.
result Asymptotic behavior of max-margin classifiers is determined by feature covariance and label covariance.