New findings show Bregman proximal algorithms can get stuck near non-stationary points.
problem Bregman proximal algorithms can get stuck near non-stationary points, misleadingly suggesting convergence.
method Analysis of Bregman proximal algorithms and their behavior near non-stationary points.
result Bregman proximal algorithms can get stuck near spurious stationary points, even in convex problems.
Paper shows no spurious local minima in a specific matrix factorization problem.
problem Optimization of ℓ1-norm rank-one symmetric matrix factorization. method Second-order variational analysis to study the landscape of the problem.
result Any second-order stationary point is globally optimal.
New study shows min-max algorithms can converge to non-stationary points.
problem Challenges in min-max optimization due to periodic cycles and spurious attractors.
method Analyzed state-of-the-art algorithms and heuristics in non-convex/non-concave problems.
result Spurious attractors can prevent min-max algorithms from reaching true optima.
Gradient descent converges geometrically to optimal self-attention parameters.
problem Training softmax self-attention layers for linear regression.
method Structure-aware gradient descent with preconditioner and regularizer.
result Gradient descent converges geometrically to global minima.
Gradient-based algorithms are effective for many machine learning tasks, but despite ample recent effort and some progress, it often remains unclear why they work in practice in optimising high-dimensional non-convex functions and why they find good minima instead of being trapped in spurious ones. Here we present a qu…
Riemannian gradient descent escapes some spurious critical points on low-rank matrix manifold.
problem Spurious critical points on the boundary of low-rank matrix manifold.
method Riemannian gradient descent with dynamical low-rank approximation and rescaled gradient flow.
result Riemannian gradient descent escapes some spurious critical points on the boundary of the manifold.
The condition for stationary increments, not scaling, detemines long time pair autocorrelations. An incorrect assumption of stationary increments generates spurious stylized facts, fat tails and a Hurst exponent H_s=1/2, when the increments are nonstationary, as they are in FX markets. The nonstationarity arises from s…
Complex-valued neural networks avoid spurious local minima.
problem Finding spurious local minima in neural networks.
method Proved no spurious local minima for shallow complex neural networks with quadratic activations.
result Complex-valued weights eliminate spurious local minima in neural networks.
The FastICA algorithm is one of the most popular iterative algorithms in the domain of linear independent component analysis. Despite its success, it is observed that FastICA occasionally yields outcomes that do not correspond to any true solutions (known as demixing vectors) of the ICA problem. These outcomes are comm…
Nonconvex matrix recovery is known to contain no spurious local minima under a restricted isometry property (RIP) with a sufficiently small RIP constant δ. If δ is too large, however, then counterexamples containing spurious local minima are known to exist. In this paper, we introduce a proof technique that is capa…
Gradient flow in phase retrieval escapes spurious minima with high probability.
problem Understanding gradient-based optimization in high-dimensional non-convex functions.
method Analytical and numerical study of gradient dynamics in phase retrieval.
result Gradient flow avoids spurious minima by drifting along unstable directions.
In this work a robust clustering algorithm for stationary time series is proposed. The algorithm is based on the use of estimated spectral densities, which are considered as functional data, as the basic characteristic of stationary time series for clustering purposes. A robust algorithm for functional data is then app…
Framework isolates causal effects from time series data, improving accuracy under non-stationarity and autocorrelation.
problem Causal inference in non-stationary, autocorrelated time series data.
method Decomposes time series into trend, seasonal, and residual components; performs component-specific causal analysis.
result Framework more accurately recovers ground-truth causal structure than state-of-the-art baselines, especially under strong non-stationarity and temporal autocorrelation.
This paper introduces SRPR for robust phase retrieval with smoothed loss functions.
problem Robust phase retrieval from noisy quadratic measurements with corruptions.
method Smoothed robust phase retrieval (SRPR) using convolution-type smoothed loss functions.
result SRPR has no spurious local solutions and benign landscape under corruptions.
Approximate vanishing ideal is a concept from computer algebra that studies the algebraic varieties behind perturbed data points. To capture the nonlinear structure of perturbed points, the introduction of approximation to exact vanishing ideals plays a critical role. However, such an approximation also gives rise to a…
The paper analyzes how good initial guesses affect the amount of data needed for low-rank matrix recovery.
problem Theoretical guarantee of local optimization algorithms requires excessive data to prevent spurious local minima.
method Quantifies the relationship between initial guess quality and sample complexity using restricted isometry constant.
result A linear improvement in initial guess quality leads to a constant factor improvement in sample complexity.
DORA analyzes deep neural networks' internal representations to detect spurious correlations.
problem Detecting spurious correlations in deep neural networks' internal representations.
method DORA uses Extreme-Activation (EA) distance measure to assess representation similarities.
result Identifies internal representations capable of detecting spurious correlations.
New method avoids spurious critical points for low-rank matrix recovery.
problem Low-rank matrix recovery problems on Riemannian manifold.
method Riemannian gradient descent with random initialization.
result Riemannian gradient descent avoids spurious critical points and converges nearly linearly.
Overparameterized models can worsen minority group errors even when overall test error improves.
problem Overparameterization exacerbates spurious correlations, harming minority groups.
method Simulations and experiments on image datasets, theoretical analysis of linear models.
result Subsampling the majority group can achieve low minority error in overparameterized models.
Study reveals different types of critical points in shallow neural networks.
problem Optimization challenges in two-layer ReLU networks with symmetry.
method Symmetry analysis, bifurcation theory, and geometric group actions.
result Different types of spurious minima have distinct loss behavior.
New method optimizes clustering with better log-likelihood landscape.
problem Nonconvex log-likelihood optimization in model-based clustering.
method Entropic optimal transport loss for Sinkhorn-EM algorithm.
result New loss function avoids spurious local optima.
New insights into spurious local minima in k-means clustering.
problem Understanding and mitigating spurious local minima in k-means clustering.
method Investigating spurious local minima under a probabilistic generative model.
result Proven structures of spurious local minima for k-means clustering.
Removing spurious features can hurt model accuracy and disproportionately affect different groups.
problem Interference from spurious features in robust model performance across different groups.
method Characterization and analysis of spurious feature removal in noiseless overparameterized linear regression.
result Removal of spurious features can decrease accuracy and disproportionately affect different groups, even in balanced datasets.
SGD quickly learns a spurious XOR feature before the signal feature, revealing learning dynamics.
problem Over-reliance on spurious correlations in neural networks trained by SGD.
method Theoretical analysis of SGD on two-layer ReLU networks trained on XOR data.
result SGD learns the spurious feature first and exponentially fast, dominating the signal feature.
This research examines rare spurious correlations in neural networks and their impact on accuracy and privacy.
problem Rare spurious correlations in neural networks and their privacy risks.
method Introducing spurious patterns correlated with a fixed class to a few training examples, analyzing ℓ2 regularization and Gaussian noise. result Rare spurious correlations can significantly impact neural network accuracy and privacy, and specific mitigation methods can be effective.
We analyze stochastic gradient algorithms for optimizing nonconvex problems. In particular, our goal is to find local minima (second-order stationary points) instead of just finding first-order stationary points which may be some bad unstable saddle points. We show that a simple perturbed version of stochastic recursiv…
New method finds points for approximating distributions faster.
problem Approximating target probability distributions using finite points.
method Stationary MMD points computed via MMD gradient flows.
result Stationary MMD points converge faster than global minimizers.
Polyconvex energies with conformal invariance have smooth stationary points outside a discrete set.
problem Stationary points of conformally invariant polyconvex energies
method Proving smoothness of stationary points
result Smooth stationary points outside a discrete set
In this paper we address the problem of solving ill-posed inverse problems in imaging where the prior is a neural generative model. Specifically we consider the decoupled case where the prior is trained once and can be reused for many different log-concave degradation models without retraining. Whereas previous MAP-bas…
Mitigates spurious correlations without bias labels.
problem Spurious correlations bias model performance.
method Introduces a novel training objective and debiasing method DPR.
result DPR achieves state-of-the-art performance.
Develops a deep non-stationary kernel for non-stationary spatio-temporal point processes.
problem Capturing non-stationary dependencies in point process data.
method Approximates the influence kernel with a novel low-rank decomposition and introduces a log-barrier penalty to maintain non-negativity.
result Demonstrates superior performance and computational efficiency compared to state-of-the-art methods.
CLIP models robustness to spurious features is re-evaluated using a new dataset.
problem Existing robustness tests of CLIP models may not fully reflect their performance on spurious features.
method Crafted a new dataset (CounterAnimal) to reveal CLIP models' reliance on realistic spurious features.
result CLIP models are robust to spurious features learned from their training data, not ImageNet.
New method finds stationary points in bilevel optimization problems.
problem Solving nonconvex-strongly-convex bilevel optimization problems.
method Restarted Accelerated HyperGradient Descent (RAHGD) method.
result Achieves best-known theoretical guarantees for finding stationary points in bilevel optimization.
Early alignment in neural networks leads to sparse representations but hinders convergence.
problem The implicit bias of gradient descent during early training phases.
method Quantitative description of early alignment phase in small initialisation, one hidden layer networks.
result Early alignment induces a sparse representation but also hinders convergence to global minima.
Study identifies and analyzes spurious correlations in data-driven models.
problem Spurious correlations in data-driven models are unreliable and hard to detect.
method Collect and analyze synthetic datasets generated from causal graphs to investigate spurious correlations.
result Patterns connecting spurious correlation hypotheses and model design choices were observed.
Self-training avoids spurious features in domain adaptation.
problem Domain shift with large differences between source and target domains.
method Entropy minimization on unlabeled target data, initialized with a source classifier.
result Entropy minimization avoids using spurious features in large domain shifts.
New algorithm finds approximate stationary points faster under differential privacy constraints.
problem Finding approximate stationary points of smooth and Lipschitz functions under differential privacy constraints.
method Developed an efficient algorithm that improves convergence rates to stationary points.
result Achieved faster rates of convergence to stationary points in both finite-sum and stochastic settings.
Nearest neighbor (k-NN) graphs are widely used in machine learning and data mining applications, and our aim is to better understand what they reveal about the cluster structure of the unknown underlying distribution of points. Moreover, is it possible to identify spurious structures that might arise due to sampling va…
This paper tackles the computational complexity of finding approximate stationary points in non-convex optimization.
problem Finding approximate stationary points in non-convex optimization problems.
method PLS-completeness, zero-order algorithms, and gradient queries.
result The query complexity of finding approximate stationary points is Θ(1/ε) for d=2.
Algorithm removes spurious concepts from neural network representations without harming task performance.
problem Spurious correlations hinder neural network out-of-distribution generalization.
method Iterative algorithm that identifies two orthogonal subspaces in neural network representation.
result Algorithm outperforms existing methods on computer vision and natural language processing benchmarks.
The paper tackles finding stationary points in stochastic convex optimization problems.
problem Finding stationary points for stochastic convex optimization problems.
method The approach relies on dimension theory to decompose the graph of the subdifferential of a convex function, showing how stochastic sampling preserves 'pieces' of these graphs, and allowing effective application of proximal-point-like methods.
result The paper provides convergence guarantees for finding stationary points in stochastic convex optimization problems.
Neural networks' weights don't converge to stationary points but training loss stabilizes.
problem The disconnect between theoretical analyses and neural network training practice.
method An invariant measure perspective inspired by ergodic theory of dynamical systems.
result The distribution of weights converges to an approximate invariant measure, explaining loss stabilization.
Study integrates ESG factors into home price predictions for U.S. cities.
problem Predicting average annual home prices using ESG factors.
method Used P-spline GAM and GLM models, transformed time series data.
result ESG factors influence home prices differently by city.
Gradient-based optimization methods are the most popular choice for finding local optima for classical minimization and saddle point problems. Here, we highlight a systemic issue of gradient dynamics that arise for saddle point problems, namely the presence of undesired stable stationary points that are no local optima…
In this article we study the regularity of stationary points of the knot energies Eα introduced by O'Hara in the range α∈(2,3). In a first step we prove that Eα is C1 on the set of all regular embedded closed curves belonging to H(α+1)/2,2 and calculate its derivative. After that we use the structure…
Paper introduces a neural network-based non-stationary influence kernel for complex event data.
problem Modeling complex, non-stationary, and dependent discrete event data.
method Neural Spectral Marked Point Processes (NSMPP) with a versatile non-stationary influence kernel.
result NSMPP outperforms state-of-the-art models on synthetic and real data.
Expectation Maximization (EM) is among the most popular algorithms for maximum likelihood estimation, but it is generally only guaranteed to find its stationary points of the log-likelihood objective. The goal of this article is to present theoretical and empirical evidence that over-parameterization can help EM avoid …
Current OOD benchmarks overestimate model robustness to spurious correlations.
problem Spurious correlations degrade OOD performance, but benchmarks show the opposite.
method Analyze OOD datasets for spurious correlations and derive conditions for robustness.
result Current OOD benchmarks are misspecified and overestimate model robustness.