This research examines anomaly detection metrics under class imbalance.
problem Challenges in interpreting evaluation metrics under class imbalance.
method Analysis of four common anomaly detection metrics (AUROC, AUPR, F1-score, MCC) under varying imbalance ratios.
result Visualisations of metric landscapes provide an intuitive view of metric preferences and stability.
MD tree diagnoses model failures using loss landscape metrics.
problem Diagnose model failures without knowing training configuration.
method MD tree based on loss landscape metrics.
result MD tree achieves 87.7% accuracy in dataset transfer tasks, outperforming validation-based approaches.
EpiMer merges models by solving Fréchet mean on a Riemannian manifold.
problem Integrating knowledge from multiple models without retraining.
method EpiMer casts model merging as solving the Fréchet mean on a Riemannian manifold, restricting computation to a low-rank subspace.
result EpiMer outperforms flat-geometry methods on image classification tasks.
New research shows flat minima in robust loss landscapes correlate with good adversarial robustness.
problem Adversarial training leads to robust overfitting, poor robust generalization.
method Average- and worst-case metrics to measure flatness in robust loss landscapes.
result Flatness in robust loss landscapes correlates with good adversarial robustness.
Persistence landscapes map persistence diagrams into a function space, which may often be taken to be a Banach space or even a Hilbert space. In the latter case, it is a feature map and there is an associated kernel. The main advantage of this summary is that it allows one to apply tools from statistics and machine lea…
Reviews recent findings on neural network landscapes.
problem Non-convexity of loss functions causing bad landscapes.
method Rigorous geometric analysis and empirical exploration.
result Wide neural nets may have sub-optimal local minima.
Study uses Bayesian Optimization to analyze noise effects in materials research.
problem Optimizing materials with many variables and experimental noise.
method Batch Bayesian Optimization with synthetic data analysis.
result Noise sensitivity varies by problem landscape, impacting optimization outcomes.
The study examines when MAML's objective has a benign landscape.
problem Understanding when MAML's objective landscape is benign.
method Analyzing the landscape of MAML objective on LQR tasks.
result The benign landscape of the MAML objective depends on task similarities.
Black holes offer insights into machine learning's loss landscapes.
problem Understanding the loss landscape in machine learning.
method Comparing machine learning loss landscapes to black hole entropy.
result Black holes provide an infinite family of potential landscapes with known minima.
Adversarial training makes logistic regression weight loss landscapes sharper.
problem Understanding why adversarial training sharpens the weight loss landscape in logistic regression.
method Theoretical analysis of linear logistic regression model with L2 norm constraints, and experiments on ResNet18.
result Adversarial training sharpens the weight loss landscape in linear logistic regression models.
AWP improves robustness by flattening weight loss landscape.
problem Improving robustness of deep neural networks against adversarial examples.
method Explicitly regularizes the flatness of weight loss landscape through adversarial weight perturbation.
result AWP forms a double-perturbation mechanism in adversarial training, leading to flatter weight loss landscape.
Efficiently infers graph edges from genetic similarity data in landscape genetics.
problem Inferring unknown graph edges from genetic similarity data in a heterogeneous landscape.
method Developed an efficient first-order optimization method to solve the inverse landscape genetics problem.
result Our method provides fast and reliable convergence, significantly outperforming existing heuristics.
Smoothed fitness landscape improves protein optimization.
problem Infeasibility of combinatorially large protein sequence space.
method Formulate protein fitness as a graph signal, smooth using Tikunov regularization, and optimize with Gibbs sampling.
result 2.5 fold fitness improvement over training set.
We construct new examples of torsional heterotic backgrounds using duality with orientifold flux compactifications. We explain how duality provides a perturbative solution to the type I/heterotic string Bianchi identity. The choice of connection used in the Bianchi identity plays an important role in the construction. …
Researchers improve visualization of neural network loss landscapes.
problem Understanding neural network generalization performance.
method Novel 'jump and retrain' procedure, non-linear dimensionality reduction (PHATE), computational homology.
result Improved visualization and quantification of neural network generalization performance.
Deeper models have a more favorable optimization landscape, making them more robust to noise.
problem Characterizing the effect of depth on the optimization landscape of linear regression models.
method Robust and over-parameterized setting, simple sub-gradient method.
result A simple sub-gradient method converges to a balanced solution that is close to the ground truth and enjoys a flat local landscape.
Machine learning techniques are being increasingly used as flexible non-linear fitting and prediction tools in the physical sciences. Fitting functions that exhibit multiple solutions as local minima can be analysed in terms of the corresponding machine learning landscape. Methods to explore and visualise molecular pot…
In topological data analysis, persistent homology is used to study the "shape of data". Persistent homology computations are completely characterized by a set of intervals called a bar code. It is often said that the long intervals represent the "topological signal" and the short intervals represent "noise". We give ev…
New sampler tackles complex discrete energy landscapes efficiently.
problem Stagnation in gradient-based discrete samplers for non-convex settings.
method DREXEL sampler with Replica Exchange and Adjusted Metropolis.
result Proves samplers satisfy detailed balance and converge to target distribution.
In most machine learning training paradigms a fixed, often handcrafted, loss function is assumed to be a good proxy for an underlying evaluation metric. In this work we assess this assumption by meta-learning an adaptive loss function to directly optimize the evaluation metric. We propose a sample efficient reinforceme…
We analyze the optimization landscape of α-loss in logistic models.
problem Optimization landscape of α-loss in logistic models.
method Tools from strictly-locally-quasi-convex functions and geometric techniques.
result Evolution of optimization landscape with respect to α.
Study reveals sharp characterisation of local minima in neural network loss landscapes.
problem Characterizing local minima in high-dimensional two-layer ReLU neural networks.
method Exact low-dimensional representation of local minima using summary statistics and link with one-pass SGD dynamics.
result Local minima in overparameterized neural networks form discrete families with varying stability and reachability.
Experimental fractal landscape dynamics observed in emulsions.
problem Understanding anomalous motions in soft glassy materials.
method Quantitative analysis of oil droplet trajectories in dense emulsions.
result Experimental fractal geometry matches computational model of soft glassy dynamics.
SGD vs quasi-Newton optimization in neural networks: different landscapes, different generalizability.
problem Understanding neural network optimization and generalizability.
method Comparison of stochastic gradient descent (SGD) and quasi-Newton optimization methods using computational tools.
result SGD solutions are separated by lower barriers than quasi-Newton solutions, but quasi-Newton solutions are deeper and more isolated.
Study visualizes actor-critic loss landscapes for inventory optimization.
problem Difficulties in solving multi-store dynamic inventory control problems.
method Low-dimensional visualizations of actor loss function.
result Loss landscapes favor optimal policies in reinforcement learning.
We study rough high-dimensional landscapes in which an increasingly stronger preference for a given configuration emerges. Such energy landscapes arise in glass physics and inference. In particular we focus on random Gaussian functions, and on the spiked-tensor model and generalizations. We thoroughly analyze the stati…
Considering the creation of persistence landscape on a parametrized curve and structure of sampling, there exists a random process for which a finite mixture model of persistence landscape (FMMPL) can provide a better description for a given dataset. In this paper, a nonparametric approach for computing integrated mean…
Analyzes adversarial training's impact on loss landscape, proposing PAS to improve model performance.
problem Challenges in optimizing models under adversarial training due to loss landscape properties.
method Analytical studies of adversarial loss functions, numerical analyses, PAS strategy.
result Adversarial training impairs optimization, but PAS strategy improves model performance.
LSAM optimizes deep learning training with improved efficiency.
problem Inefficiency in distributed large-batch training with Sharpness-Aware Minimization (SAM).
method Integrates SAM's adversarial steps with an asynchronous distributed sampling strategy.
result Higher final accuracy compared to data-parallel SAM.
Paper characterizes optimization landscape of Tucker decomposition.
problem Finding exact Tucker decomposition is a nonconvex optimization problem.
method Characterized the optimization landscape and provided a local search algorithm.
result All local minima are globally optimal if tensor has an exact Tucker decomposition.
Researchers improve NCE by addressing its flat loss landscape issues.
problem NCE's poor performance due to an ill-behaved loss landscape.
method Introduced eNCE with an exponential loss and normalized gradient descent.
result Proven that landscape issues arise from inappropriate noise distribution.
Embedding principle explains loss landscape of deep neural networks.
problem Understanding the structure of loss landscapes in deep neural networks.
method Proposed an embedding principle that critical points of narrower DNNs can be embedded to critical points of wider DNNs.
result Wide DNNs are often attracted by highly-degenerate critical points embedded from narrower DNNs.
There are many surprising and perhaps counter-intuitive properties of optimization of deep neural networks. We propose and experimentally verify a unified phenomenological model of the loss landscape that incorporates many of them. High dimensionality plays a key role in our model. Our core idea is to model the loss la…
TDA-based portfolios show better risk-adjusted returns than classical methods.
problem Traditional portfolio selection methods fail to capture complex asset dynamics.
method Topological Data Analysis (TDA) using persistence landscapes to quantify portfolio risk.
result TDA-based portfolios outperform classical models in excess mean return and financial ratios.
Deep learning dynamics and NTK evolution studied through diverse measures.
problem Understanding the training dynamics of deep neural networks and their loss landscapes.
method Phenomenological analysis of training dynamics in multiple architectures and datasets.
result Training dynamics exhibit a chaotic initial transient followed by a stable phase, with the NTK evolving to match full network performance.
The landscape of empirical risk has been widely studied in a series of machine learning problems, including low-rank matrix factorization, matrix sensing, matrix completion, and phase retrieval. In this work, we focus on the situation where the corresponding population risk is a degenerate non-convex loss function, nam…
In this paper we first analyzed the inductive bias underlying the data scattered across complex free energy landscapes (FEL), and exploited it to train deep neural networks which yield reduced and clustered representation for the FEL. Our parametric method, called Information Distilling of Metastability (IDM), is end-t…
The study tests inferences about neural network optimization from linear interpolation of loss landscapes.
problem Understanding the difficulty of neural network optimization problems.
method Linear interpolation of neural network loss landscapes, systematic evaluation of various factors.
result Linear interpolation does not correlate with model performance, challenging prior intuition.
The paper analyzes phase retrieval under limited samples, ensuring a benign local landscape for convergence.
problem Ensuring a benign local landscape for phase retrieval under limited samples.
method Fine-grained analysis of local landscape properties under the regime of limited samples.
result Gradient descent can converge to an od(1)-loss solution exponentially fast under certain conditions. Deep ReLU networks with extra parameters have mostly good loss landscapes.
problem Finding good local minima in the loss landscape of deep neural networks.
method Analyzing shallow and deep ReLU networks with extra parameters on a generic dataset.
result Most activation patterns correspond to regions with no bad local minima.
We investigate the structure of the profit landscape obtained from the most basic, fluctuation based, trading strategy applied for the daily stock price data. The strategy is parameterized by only two variables, p and q. Stocks are sold and bought if the log return is bigger than p and less than -q, respectively. Repet…
This work shows how to efficiently simulate parts of quantum landscapes using classical computers.
problem Identifying where quantum computers are advantageous and offloading computations.
method Developed a quantum-enhanced classical algorithm to simulate sub-regions of quantum landscapes.
result It is possible to generate a classical surrogate of a sub-region of a quantum landscape.
Study reveals pathological eigenvalue spectra in FIM and its variants of DNNs.
problem Understanding sharp local shapes in DNN loss landscapes.
method Analysis of FIM and its variants in regression and classification DNNs.
result Pathological eigenvalue spectra appear in FIM and its variants, indicating sharp local shapes in specific directions.
This work justifies neural collapse under MSE loss and analyzes the optimization landscape.
problem Understanding neural collapse in deep neural networks under MSE loss.
method Global landscape analysis of vanilla nonconvex MSE loss.
result The only global minimizers are neural collapse solutions.
Geodesics connect model modes in neural network loss landscapes.
problem Connecting modes in neural network loss landscapes.
method Reframed mode connectivity in Information Geometry, hypothesized geodesics as mode-connecting paths, proposed algorithm to approximate geodesics.
result Geodesics achieve mode connectivity in neural networks.
Model explains surprising properties of neural network training landscapes.
problem Understanding the geometry of neural network loss landscapes.
method Developed a simple theoretical model of gradients and Hessians.
result Unified model accounts for 4 surprising properties of neural loss landscapes.
New method simplifies optimization landscapes by transforming saddle points.
problem Saddle points hinder non-convex optimization in machine learning.
method Variable elimination algorithms, like VarPro, are compared to reveal geometric insights.
result Variable elimination reshapes critical point structure, creating local maxima from saddle points.
Realistic evolutionary fitness landscapes are notoriously difficult to construct. A recent cutting-edge model of virus assembly consists of a dodecahedral capsid with 12 corresponding packaging signals in three affinity bands. This whole genome/phenotype space consisting of 312 genomes has been explored via comp…