This paper improves forecast stability without sacrificing accuracy using dynamic loss weighting.
problem Rolling origin forecast instability in time series forecasting.
method Dynamic loss weighting algorithms applied to the N-BEATS model.
result Dynamic loss weighting can further improve forecast stability without compromising accuracy.
HydaLearn dynamically adjusts task weights for better MTL performance.
problem Constant loss weights in MTL lead to poor results due to drifting relevance and varying mini-batch composition.
method HydaLearn uses mini-batch gradients to dynamically adjust task weights.
result HydaLearn improves performance on synthetic and real-world data.
SoftAdapt dynamically adjusts loss weights for multi-part functions.
problem Slow convergence and poor weight selection for multi-part loss functions.
method SoftAdapt dynamically changes weights based on live performance statistics.
result Improved convergence and better weight selection for multi-part loss functions.
Multi-task learning is a method for improving the generalizability of multiple tasks. In order to perform multiple classification tasks with one neural network model, the losses of each task should be combined. Previous studies have mostly focused on multiple prediction tasks using joint loss with static weights for tr…
New loss function helps learn unstable dynamical systems.
problem Gradient descent fails to learn unstable dynamical systems.
method Introduced a time-weighted logarithmic loss function.
result Time-weighted loss function effectively learns unstable systems.
SLERP interpolation optimizes dynamic weight rebalancing in AMMs.
problem Optimizing dynamic weight rebalancing in automated market makers (AMMs).
method Riemannian geometry and SLERP interpolation.
result SLERP interpolation minimizes the KL divergence loss in dynamic weight rebalancing.
A framework for multi-label sentiment analysis in 100 languages with dynamic weighting.
problem Cross-lingual sentiment analysis in multi-label settings with label imbalance.
method Dynamic weighting method, focal loss adaptation, optimal class-specific thresholds.
result State-of-the-art performance in 7 out of 9 metrics across 3 languages.
Neural networks' weights don't converge to stationary points but training loss stabilizes.
problem The disconnect between theoretical analyses and neural network training practice.
method An invariant measure perspective inspired by ergodic theory of dynamical systems.
result The distribution of weights converges to an approximate invariant measure, explaining loss stabilization.
Comment on entropy learning for dynamic treatment regimes.
problem Evaluating dynamic treatment regimes using entropy loss.
method Optimization-based alternative to IPW estimate.
result Suggests optimization-based approach for evaluation.
Theoretical justification for deep networks' performance with regularization techniques.
problem Understanding the performance of deep networks trained with the square loss.
method Analysis of gradient flow and theoretical justification of regularization techniques.
result Convergence to solutions with smaller Frobenius norms leads to better classification error bounds.
We propose a simple discrete time semi-supervised graph embedding approach to link prediction in dynamic networks. The learned embedding reflects information from both the temporal and cross-sectional network structures, which is performed by defining the loss function as a weighted sum of the supervised loss from past…
Improved loss functions adapt to weight-space anisotropy, outperforming isotropic counterparts.
problem Adapting to the anisotropic nature of deep weight spaces for better performance.
method Refined local entropic loss functions restricted to a subset of weights, exploiting anisotropy.
result Partial local entropies outperform isotropic counterparts on image classification tasks.
The paper studies neural networks' convergence near origin and saddle points.
problem Directional convergence of neural networks near small initializations and saddle points.
method Gradient flow dynamics analysis of two-homogeneous neural networks.
result Neural networks' weights approximately converge in direction to KKT points for small initializations.
Weight decay stabilizes training dynamics by slowing progressive sharpening.
problem Understanding how weight decay affects training stability in deep learning models.
method Analyzing weight decay effects at the Edge of Stability, developing a mathematical framework.
result Weight decay dampens oscillations and stabilizes sharpness in CNNs, causing a phase transition in MLPs.
MTAdam optimizes multiple loss terms in neural models, balancing gradients dynamically.
problem Balancing multiple loss terms in neural model training is challenging and computationally demanding.
method Generalized Adam algorithm that computes separate derivatives and balances gradients across layers dynamically.
result Training with MTAdam leads to faster recovery from suboptimal initial loss weighting and matches conventional training outcomes.
MANA-Net improves market predictions by dynamically weighting news sentiments.
problem Aggregated Sentiment Homogenization in financial news data.
method Dynamic market-news attention mechanism to aggregate sentiments.
result MANA-Net outperforms recent market prediction methods by 1.1% Profit & Loss and 0.252 daily Sharpe ratio.
This paper reformulates Fβ for better model performance and interpretation.
problem Optimizing model performance and interpretation using Fβ metric. method Reformulate Fβ metric to facilitate statistical distributions and dynamic penalty weights. result Better and interpretable results with a 14% boost in F1 score for IMDB data. Constraints improve deep neural network training by stabilizing and enhancing robustness.
problem Vanishing/exploding gradients and poor weight magnitudes in deep neural networks.
method Weight-constrained stochastic dynamics using Langevin dynamics framework.
result Enhanced exploration of the loss landscape and improved generalization.
Dynamic-weight AMMs outperform traditional CEX rebalancing in tokenized funds, especially on L2s.
problem Improving asset allocation efficiency in decentralized finance (DeFi) protocols.
method Block-level arbitrage analysis and long-term performance benchmarks on two live pools.
result Dynamic-weight AMMs can achieve performance comparable to or better than traditional CEX rebalancing, especially on Layer 2 (L2) networks.
Gradient flow with weight decay shows grokking effect in deep learning.
problem Understanding the grokking effect in deep learning.
method Analyzing gradient flow dynamics with weight decay.
result Weight decay causes slow norm reduction, explaining grokking.
The study improves VaR forecast accuracy by modeling conditional quantile dynamics.
problem Improving the accuracy of Value-at-Risk (VaR) forecasts for time-varying quantiles.
method Time-varying modeling of VaR, evaluation via simulation, asymmetric Mean Absolute Deviation loss function.
result Substantial improvements in forecasting conditional quantiles by maintaining predicted quantile unchanged.
DP-Net uses dynamic programming for efficient deep neural network compression.
problem Efficiently compressing deep neural networks while maintaining accuracy.
method Dynamic Programming for optimal weight quantization and clustering-friendly training.
result Achieves up to 77X compression ratio on Wide ResNet with minimal accuracy loss.
The paper introduces a multi-step loss function to improve model-based reinforcement learning.
problem Compounding of one-step prediction errors in long trajectories.
method A multi-step objective function combining MSE losses at various future horizons.
result Models trained with the multi-step loss achieve significant improvement in future prediction.
AWP improves robustness by flattening weight loss landscape.
problem Improving robustness of deep neural networks against adversarial examples.
method Explicitly regularizes the flatness of weight loss landscape through adversarial weight perturbation.
result AWP forms a double-perturbation mechanism in adversarial training, leading to flatter weight loss landscape.
Framework for differentiating WFSTs for structured loss functions.
problem Training dynamic structured loss functions in neural networks.
method Automatic differentiation with WFSTs, combining pruning and back-off.
result Demonstrated learning over WFST latent phrase decomposition.
Near-logarithmic regret per switch achieved for mixable/exp-concave losses.
problem Online optimization of mixable loss functions with dynamic environments.
method Online mixture framework using static solvers and hyper-expert creations.
result Near-logarithmic regret per switch with sub-polynomial complexity.
Improves transfer learning by weighting importance based on test-over-training density.
problem Distribution shift in training and test data.
method Joint and dynamic importance-predictor estimation, causal mechanism transfer.
result Enhanced transfer learning performance in complex, high-dimensional tasks.
Adversarial training makes logistic regression weight loss landscapes sharper.
problem Understanding why adversarial training sharpens the weight loss landscape in logistic regression.
method Theoretical analysis of linear logistic regression model with L2 norm constraints, and experiments on ResNet18.
result Adversarial training sharpens the weight loss landscape in linear logistic regression models.
We reparametrize ReLU NNs as splines to understand their learning dynamics.
problem Understanding the learning dynamics and inductive bias of neural networks.
method Reparametrize ReLU NNs as continuous piecewise linear splines to study learning dynamics.
result Standard weight initializations yield very flat functions, leading to strength and type of implicit regularization.
One-pass SGD dynamics in overparameterized quadratic networks show slow escape from poor solutions.
problem Slow escape from poor generalization solutions in overparameterized neural networks.
method Analysis of one-pass SGD dynamics using ordinary differential equations for overlap matrices.
result Overparameterization only modestly accelerates escape from poor solutions.
A new method improves flow matching by dynamically weighting density estimates.
problem High-dimensional integration inefficiency in flow matching.
method Density-weighted Dynamic Stein operators.
result Significant improvement in vector field smoothness and sampling efficiency.
New insights into CE dynamics reveal how Hadamard initialization simplifies softmax.
problem Understanding the dynamics of cross-entropy training loss in deep learning.
method Analyzing a two-layer linear neural network with standard-basis vectors as inputs.
result Gradient flow on cross-entropy converges to neural collapse geometry, proving global convergence.
Though machine learning algorithms excel at minimizing the average loss over a population, this might lead to large discrepancies between the losses across groups within the population. To capture this inequality, we introduce and study a notion we call maximum weighted loss discrepancy (MWLD), the maximum (weighted) d…
Paper proves stability of multi-dimensional rarefaction waves in gas dynamics.
problem Challenges in constructing multi-dimensional rarefaction waves in gas dynamics.
method Geometric Weighted Energy Method (GWEM) to overcome derivative losses.
result Established nonlinear stability of multi-dimensional rarefaction waves for compressible Euler equations.
We study the problem of training deep neural networks with Rectified Linear Unit (ReLU) activation function using gradient descent and stochastic gradient descent. In particular, we study the binary classification problem and show that for a broad family of loss functions, with proper random weight initialization, both…
Optimization algorithms help overparameterized neural networks achieve high performance.
problem Understanding the convergence of optimization algorithms on overparameterized neural networks.
method Analyzing a broad class of optimization algorithms using dynamical systems and finite over-parameterized neural networks with ReLU activation.
result The Heavy Ball method converges to global minimum at a linear rate, while NAG converges sublinearly.
We consider the problem of estimating a low-rank matrix from a noisy observed matrix. Previous work has shown that the optimal method depends crucially on the choice of loss function. In this paper, we use a family of weighted loss functions, which arise naturally for problems such as submatrix denoising, denoising wit…
The paper identifies five extreme learning regimes for large linear autoencoders.
problem Understanding the learning dynamics of large weight-tied linear autoencoders.
method Formal loss-expansion hierarchy and analysis of gradient flow.
result Five extreme regimes associated with faces of a triangular prism.
To improve the execution speed and efficiency of neural networks in embedded systems, it is crucial to decrease the model size and computational complexity. In addition to conventional compression techniques, e.g., weight pruning and quantization, removing unimportant activations can reduce the amount of data communica…
Derives equations for deep learning biases and weights, showing data complexity reduction.
problem Understanding interpretability in supervised learning.
method Gradient flow equations and dynamical truncation of training data.
result Data complexity reduction at an exponential rate with training.
Motivated by the asset-liability management of a nuclear power plant operator, we consider the problem of finding the least expensive portfolio, which outperforms a given set of stochastic benchmarks. For a specified loss function, the expected shortfall with respect to each of the benchmarks weighted by this loss func…
This research shows loss weighting remains effective in last layer retraining despite model overparameterization.
problem Overcoming biases in machine learning models at scale.
method Theoretical and practical exploration of last layer retraining in an overparameterized setting.
result Loss weighting is still effective in last layer retraining, but weights must account for model overparameterization.
New loss function restores importance weighting in overparameterized models.
problem Restoring importance weighting in overparameterized neural networks.
method Introduced polynomially-tailed losses to restore effects of importance weighting.
result Polynomially-tailed losses improve performance in correcting distribution shift.
Much of the focus in machine learning research is placed in creating new architectures and optimization methods, but the overall loss function is seldom questioned. This paper interprets machine learning from a multi-objective optimization perspective, showing the limitations of the default linear combination of loss f…
Enhances linear regression with Kalman filter for loss minimization.
problem Minimizing loss in linear regression models.
method Integrates Kalman filter and SGD for optimal weight updates.
result Develops optimal linear regression equation with minimum area under curve.
It is essential to incorporate the impact of investor behavior when modeling the dynamics of asset returns. In this paper, we reconcile behavioral finance and rational finance by incorporating investor behavior within the framework of dynamic asset pricing theory. To include the views of investors, we employ the method…
We explore how neural networks train to zero loss, focusing on initial scale.
problem Understanding neural network training dynamics and zero loss.
method Macroscopic limits analysis of gradient descent dynamics.
result Gradient descent can drive deep neural networks to zero loss regardless of initialization.
A new method lifts training of input-convex neural networks to avoid dead weights and plateaued loss.
problem Training input-convex neural networks with non-negative weights.
method Introduces a hypernetwork that emits non-negative weights from a summary of the input batch, adding stochasticity to soften the loss landscape.
result The lift method achieves lower test loss than projected gradient descent and direct softplus reparametrization.