In this paper, we study the implicit regularization of the gradient descent algorithm in homogeneous neural networks, including fully-connected and convolutional neural networks with ReLU or LeakyReLU activations. In particular, we study the gradient descent or gradient flow (i.e., gradient descent with infinitesimal s…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “homogeneous loss”
Homogenized SGD explains SGD dynamics in high dimensions.
problem Understanding SGD dynamics in high-dimensional settings.
method Developed a homogenized SGD model to analyze high-dimensional SGD.
result Convergent high-dimensional SGD to homogenized SGD for quadratic statistics.
With an eye toward understanding complexity control in deep learning, we study how infinitesimal regularization or gradient descent optimization lead to margin maximizing solutions in both homogeneous and non-homogeneous models, extending previous work that focused on infinitesimal regularization only in homogeneous mo…
Large GD stepsizes improve margins and speed up training for non-homogeneous networks.
problem Training efficiency and margin improvement in non-homogeneous two-layer networks.
method Investigation of two distinct phases in GD training, showing margin growth and empirical risk decrease.
result Large GD stepsizes lead to faster convergence and improved margins in non-homogeneous networks.
GD iterates for non-homogeneous deep nets increase margin and converge in direction.
problem Understanding implicit bias in non-homogeneous deep networks.
method Characterization of GD iterates' properties starting from small empirical risk.
result GD iterates converge in direction despite diverging norms, satisfying KKT conditions.
Directional Convergence Near Small Initializations and Saddles in Two-Homogeneous Neural Networkscs.LG
The paper studies neural networks' convergence near origin and saddle points.
problem Directional convergence of neural networks near small initializations and saddle points.
method Gradient flow dynamics analysis of two-homogeneous neural networks.
result Neural networks' weights approximately converge in direction to KKT points for small initializations.
This work analyzes the maximum-margin bias in quasi-homogeneous neural networks.
problem Analyzing the maximum-margin bias in quasi-homogeneous neural networks.
method Geometric analysis of gradient dynamics for quasi-homogeneous models.
result Gradient flow implicitly favors a subset of parameters, leading to asymmetric norm minimization.
Neural network approach simplifies multiscale problem homogenization.
problem Homogenizing multiscale problems with varying microscale structures.
method Derivative-free neural network with Brownian walkers.
result Neural network method is computationally efficient and robust.
This paper examines the uniform properties of AMMs in cryptocurrency markets.
problem Theoretical uniformity of AMMs despite diverse strategies.
method Derives a universal formula for liquidity provisioning and compares models.
result Constant function and token swap models are equivalent under uniform liquidity.
Gradient descent is a simple and widely used optimization method for machine learning. For homogeneous linear classifiers applied to separable data, gradient descent has been shown to converge to the maximal margin (or equivalently, the minimal norm) solution for various smooth loss functions. The previous theory does …
DNNs with regularization reveal feature learning dynamics and sparsity.
problem Understanding feature learning in DNNs with regularization.
method Reformulating loss in terms of layerwise activations and covariances.
result Proving sparsity of local minima in -regularized DNNs.
New algorithm tackles heterogeneous curvature in online convex optimization.
problem Adversarial bandit convex optimization with varying curvature.
method Developed an adaptive algorithm that learns curvature on the fly.
result Achieves optimal regret bounds even with heterogeneous curvature.
The paper connects flatness to generalization in learning multi-index models with neural networks.
problem Understanding the generalization of non-convex neural networks using flatness measures.
method Analyzes 2-layer non-convex homogeneous neural networks and their connection to multi-index models.
result Flattest interpolators achieve small population loss and generalize well, establishing a direct link between flatness and generalization.
Paper proposes a method to estimate counterfactual outcomes without a known SCM.
problem Estimating counterfactual outcomes without a known structural causal model.
method Introduces rank preservation assumption and a novel ideal loss for unbiased learning of counterfactual outcomes.
result The proposed method is effective and unbiased, as shown by theoretical analysis and experiments.
We examine gradient descent on unregularized logistic regression problems, with homogeneous linear predictors on linearly separable datasets. We show the predictor converges to the direction of the max-margin (hard margin SVM) solution. The result also generalizes to other monotone decreasing loss functions with an inf…
We present a fully-supervized method for learning to segment data structured by an adjacency graph. We introduce the graph-structured contrastive loss, a loss function structured by a ground truth segmentation. It promotes learning vertex embeddings which are homogeneous within desired segments, and have high contrast …
Study on risk contributions of portfolios using lambda quantile risk measures.
problem No known allocation rule for non-positively homogeneous risk measures.
method Defined lambda quantiles on portfolio compositions, derived derivatives, and introduced generalized Euler contributions.
result Explicit formulae for the derivatives of lambda quantiles, showing their homogeneity properties.
The late-stage training dynamics of (stochastic) subgradient descent on homogeneous neural networkscs.LG
SGD converges to critical points of normalized margin in late-stage training for homogeneous neural networks.
problem Analyzing the implicit bias of SGD on homogeneous neural networks.
method Interpreting SGD dynamics as an Euler-like discretization of a conservative field flow associated with the normalized classification margin.
result Normalized SGD iterates converge to the set of critical points of the normalized margin at late-stage training.
We embed KKT points in neural networks of different sizes.
problem Classifying data using homogeneous neural networks.
method Introducing KKT point embedding principle and proving it for different network types.
result KKT points of a smaller network can be mapped to those of a larger network via linear transformations.
We study the impact of contagion in a network of firms facing credit risk. We describe an intensity based model where the homogeneity assumption is broken by introducing a random environment that makes it possible to take into account the idiosyncratic characteristics of the firms. We shall see that our model goes behi…
Gradient descent on normalized networks reveals sparsity preferences.
problem Understanding the inductive bias of gradient descent on normalized neural nets.
method Analysis of gradient descent on weight-normalized smooth homogeneous neural nets, focusing on SWN and EWN.
result EWN causes weights to be updated in a way that prefers asymptotic relative sparsity.
AEGCN uses autoencoder constraints to improve graph node classification.
problem Node classification on graph domains with reduced information loss.
method Autoencoder-constrained graph convolutional network (AEGCN).
result Adding autoencoder constraints significantly improves graph convolutional network performance.
A new combinatorial approach groups regression coefficients for improved accuracy.
problem Grouping regression coefficients to reveal shared values within groups.
method Introduces -Fusion, a combinatorial grouping approach using mixed integer optimization.
result -Fusion achieves grouping consistency under weak grouping sensitivity conditions.
Stochastic Gradient Descent (SGD) is a central tool in machine learning. We prove that SGD converges to zero loss, even with a fixed (non-vanishing) learning rate - in the special case of homogeneous linear classifiers with smooth monotone loss functions, optimized on linearly separable data. Previous works assumed eit…
Improved computational complexity in statistical models using second-order information.
problem Polynomial convergence of gradient descent in singular statistical models.
method Normalized Gradient Descent (NormGD) algorithm with second-order information.
result NormGD reaches final statistical radius in logarithmic iterations of .
New research shows logistic regression can achieve optimal error rate for agnostic learning of halfspaces.
problem Agnostic learning of homogeneous halfspaces with logistic loss.
method Constructing a well-behaved distribution and using logistic regression with additional convex optimization steps.
result Logistic regression can achieve misclassification risk, matching the upper bound.
Symmetry in loss functions constrains model parameters, leading to specific learning outcomes.
problem Understanding and leveraging symmetries in neural networks to improve learning outcomes.
method Analyzing the impact of loss function symmetries on model parameters and learning behavior.
result Mirror-reflection symmetries in loss functions lead to constraints on model parameters, influencing learning outcomes.
New test for SGD in binary classification reduces computation time.
problem Determining optimal stopping for SGD in binary classification.
method Proposes a new, simple, computationally inexpensive termination criterion for SGD.
result Termination criterion reduces expected misclassification probability.
Unpaired image-to-image translation has attracted significant interest due to the invention of CycleGAN, a method which utilizes a combination of adversarial and cycle consistency losses to avoid the need for paired data. It is known that the CycleGAN problem might admit multiple solutions, and our goal in this paper i…
Heavy-tailed distributions emerge in SGD's parameter evolution.
problem Understanding heavy-tailed distributions in SGD parameter evolution.
method Continuous diffusion approximation of SGD (homogenized SGD) analysis.
result Explicit upper and lower bounds on tail-index of homogenized SGD.
Numerical challenges inherent in algorithms for computing worst Value-at-Risk in homogeneous portfolios are identified and solutions as well as words of warning concerning their implementation are provided. Furthermore, both conceptual and computational improvements to the Rearrangement Algorithm for approximating wors…
Study selective classification with halfspaces, achieving error bounds under Gaussian distributions.
problem Modeling relationships in subsets of data defined by selection rules.
method Sparse linear classifiers for subsets defined by halfspaces, focusing on Gaussian feature distributions.
result First PAC-learning algorithm for homogeneous halfspace selectors with error guarantee $\bigO*{\sqrt{\mathrm{opt}}}$.
In structural credit risk models, default events and the ensuing losses are both derived from the asset values at maturity. Hence it is of utmost importance to choose a distribution for these asset values which is in accordance with empirical data. At the same time, it is desirable to still preserve some analytical tra…
In this paper we study the effect of network structure between agents and objects on measures for systemic risk. We model the influence of sharing large exogeneous losses to the financial or (re)insuance market by a bipartite graph. Using Pareto-tailed losses and multivariate regular variation we obtain asymptotic resu…
Generalized Orlicz premiaq-fin.RM
Introduces generalized Orlicz premia for broader applicability.
problem Developing a flexible framework for insurance premium calculation.
method Introduces a generalized Orlicz premium definition using non-convex loss functions.
result Generalized Orlicz premia encompass various specific cases and maintain key properties.
It is a well known fact that recovery rates tend to go down when the number of defaults goes up in economic downturns. We demonstrate how the loss given default model with the default and recovery dependent via the latent systematic risk factor can be estimated using Bayesian inference methodology and Markov chain Mont…
This paper quantifies and mitigates a bias in the Hayashi-Yoshida estimator causing data loss.
problem Formulaic bias in the Hayashi-Yoshida estimator leading to data loss.
method Formalizes and quantifies the data loss, introduces (a,b)-asynchronous adversary, and provides algorithms.
result Proves that for equal rates, the minimal average cumulative data loss is 25%.
Catapult phase in neural nets shows exponential loss growth before quick decrease.
problem Understanding phase transitions in neural networks during training.
method Analyzing weight norm and loss behavior for super-critical learning rates.
result Proven existence of catapult phase in quadratic models and two-layer nets.
Under the Basel II standards, the Operational Risk (OpRisk) advanced measurement approach allows a provision for reduction of capital as a result of insurance mitigation of up to 20%. This paper studies the behaviour of different insurance policies in the context of capital reduction for a range of possible extreme los…
Introduces new types of homogeneous spaces and their properties.
problem Defining and understanding new types of homogeneous spaces.
method Introducing and analyzing (strongly) (-)discrete homogeneous spaces.
result Discovers relationships between new and existing homogeneous space types.
Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Lossmath.OC
Analysis of gradient descent on wide neural networks reveals strong generalization.
problem Understanding why wide neural networks trained with logistic loss perform well.
method Characterization of gradient flow limits and comparison to max-margin classifier.
result Margin is independent of ambient dimension, leading to strong generalization.
The paper examines the unexpected losses and risk ratios for co-monotonic alternatives in large portfolios.
problem Understanding the unexpected losses and risk ratios for large portfolios with co-monotonic alternatives.
method Analyzes the asymptotic behavior of unexpected losses and risk ratios for co-monotonic alternatives using monotone cash-additive risk measures and Choquet insurance premia.
result Unexpected losses of large weighted portfolios are of order , where is the average weight.
A simple banking network model is proposed which features multiple waves of bank defaults and is analytically solvable in the limiting case of an infinitely large homogeneous network. The model is a collection of nodes representing individual banks; associated with each node is a balance sheet consisting of assets and …
Graph node embedding aims at learning a vector representation for all nodes given a graph. It is a central problem in many machine learning tasks (e.g., node classification, recommendation, community detection). The key problem in graph node embedding lies in how to define the dependence to neighbors. Existing approach…
We develop an agent-based simulation of the catastrophe insurance and reinsurance industry and use it to study the problem of risk model homogeneity. The model simulates the balance sheets of insurance firms, who collect premiums from clients in return for ensuring them against intermittent, heavy-tailed risks. Firms m…
MANA-Net improves market predictions by dynamically weighting news sentiments.
problem Aggregated Sentiment Homogenization in financial news data.
method Dynamic market-news attention mechanism to aggregate sentiments.
result MANA-Net outperforms recent market prediction methods by 1.1% Profit & Loss and 0.252 daily Sharpe ratio.
Deep learning clusters patient time-series data for better prognosis.
problem Clustering time-series data for patient phenotyping and prognosis.
method Deep predictive clustering with novel loss functions for future outcome distribution.
result Model achieves superior clustering performance and identifies meaningful patient subgroups.
Introduces homogeneity supermanifolds for studying graded structures.
problem Graded structures on supermanifolds.
method Homogeneity degrees and weight vector field.
result Proofs of homogeneous Poincaré Lemma, Frobenius Theorem, and Darboux Theorem.