The paper examines GANs' forgetting and mode collapse, showing how they relate and impact training.
problem Catastrophic forgetting and mode collapse in GANs during continual learning.
method Investigates the continual learning nature of GANs, analyzing discriminator's output landscapes and convergence.
result Catastrophic forgetting and mode collapse are interrelated and prevent GANs from converging.
Study analyzes landscape complexity of empirical loss functions with correlated data.
problem Understanding the complexity of loss landscapes in machine learning with structured data.
method Kac-Rice formula and random matrix theory applied to high-dimensional empirical loss functions.
result Characterizes the average number of critical points in loss functions with structured data.
Improved GAN sampling by collaborating discriminator and generator.
problem Loss of discriminator information during standard GAN sampling.
method Collaborative sampling between generator and discriminator, gradient-based updates.
result Generated samples are closer to real data distribution.
Hybrid method improves mutual information estimation from samples.
problem Estimating mutual information from joint distributions is challenging.
method Proposes a hybrid method combining discriminative and generative approaches.
result Hybrid methods yield tighter variational bounds on mutual information.
We introduce the Probabilistic Generative Adversarial Network (PGAN), a new GAN variant based on a new kind of objective function. The central idea is to integrate a probabilistic model (a Gaussian Mixture Model, in our case) into the GAN framework which supports a new kind of loss function (based on likelihood rather …
Enhances deep learning robustness to noise without sacrificing clean data accuracy.
problem Robustness of deep neural networks to input noise.
method Discriminative loss at penultimate layer and class-wise feature alignment with Gaussian noise.
result Improves robustness to various perturbations without degrading clean data accuracy.
Persistence landscapes map diagrams into function spaces for statistical and machine learning applications.
problem Mapping persistence diagrams into function spaces for statistical and machine learning.
method Introducing persistence landscapes, weighted persistence landscapes, and Poisson-weighted persistence landscape kernels.
result Persistence landscapes allow for the application of statistical and machine learning tools, and are stable and invertible.
Reviews recent findings on neural network landscapes.
problem Non-convexity of loss functions causing bad landscapes.
method Rigorous geometric analysis and empirical exploration.
result Wide neural nets may have sub-optimal local minima.
The paper analyzes RfQ processes on MD2C platforms using probabilistic models.
problem Analyzing the negotiation process on MD2C platforms to ensure dealers' profitability.
method Probabilistic graphical models and causal inference.
result Generative models can match predictive accuracy of discriminative models while enforcing business requirements.
PropEn uses matching to create a larger dataset for efficient design optimization.
problem Limited data and complex landscapes in scientific applications.
method PropEn uses a matching approach to implicitly guide design without a discriminator.
result PropEn efficiently approximates the gradient of property improvement within the data distribution.
The study examines when MAML's objective has a benign landscape.
problem Understanding when MAML's objective landscape is benign.
method Analyzing the landscape of MAML objective on LQR tasks.
result The benign landscape of the MAML objective depends on task similarities.
Black holes offer insights into machine learning's loss landscapes.
problem Understanding the loss landscape in machine learning.
method Comparing machine learning loss landscapes to black hole entropy.
result Black holes provide an infinite family of potential landscapes with known minima.
Adversarial training makes logistic regression weight loss landscapes sharper.
problem Understanding why adversarial training sharpens the weight loss landscape in logistic regression.
method Theoretical analysis of linear logistic regression model with L2 norm constraints, and experiments on ResNet18.
result Adversarial training sharpens the weight loss landscape in linear logistic regression models.
AWP improves robustness by flattening weight loss landscape.
problem Improving robustness of deep neural networks against adversarial examples.
method Explicitly regularizes the flatness of weight loss landscape through adversarial weight perturbation.
result AWP forms a double-perturbation mechanism in adversarial training, leading to flatter weight loss landscape.
Efficiently infers graph edges from genetic similarity data in landscape genetics.
problem Inferring unknown graph edges from genetic similarity data in a heterogeneous landscape.
method Developed an efficient first-order optimization method to solve the inverse landscape genetics problem.
result Our method provides fast and reliable convergence, significantly outperforming existing heuristics.
Machine learning speeds up the construction of virus assembly fitness landscapes.
problem Constructing realistic evolutionary fitness landscapes for viruses is computationally expensive.
method Developed a neural network to model virus assembly efficiency from a whole genome/phenotype space.
result Machine learning significantly reduces the computational time for constructing fitness landscapes.
Smoothed fitness landscape improves protein optimization.
problem Infeasibility of combinatorially large protein sequence space.
method Formulate protein fitness as a graph signal, smooth using Tikunov regularization, and optimize with Gibbs sampling.
result 2.5 fold fitness improvement over training set.
Unified model predicts structure of neural network loss landscapes.
problem Understanding the structure of neural network loss landscapes.
method Modeling loss landscape as high-dimensional wedges, analyzing hyperparameters' effects.
result Existence of low-loss subspaces connecting solutions.
Researchers improve visualization of neural network loss landscapes.
problem Understanding neural network generalization performance.
method Novel 'jump and retrain' procedure, non-linear dimensionality reduction (PHATE), computational homology.
result Improved visualization and quantification of neural network generalization performance.
Deeper models have a more favorable optimization landscape, making them more robust to noise.
problem Characterizing the effect of depth on the optimization landscape of linear regression models.
method Robust and over-parameterized setting, simple sub-gradient method.
result A simple sub-gradient method converges to a balanced solution that is close to the ground truth and enjoys a flat local landscape.
Machine learning techniques are being increasingly used as flexible non-linear fitting and prediction tools in the physical sciences. Fitting functions that exhibit multiple solutions as local minima can be analysed in terms of the corresponding machine learning landscape. Methods to explore and visualise molecular pot…
New sampler tackles complex discrete energy landscapes efficiently.
problem Stagnation in gradient-based discrete samplers for non-convex settings.
method DREXEL sampler with Replica Exchange and Adjusted Metropolis.
result Proves samplers satisfy detailed balance and converge to target distribution.
Paper presents a more accurate method for nonparametric density estimation using FMMPL and SIR.
problem Improving nonparametric density estimation for complex datasets.
method Finite mixture model of nonparametric density estimation using sampling importance resampling.
result FMMPL provides more accurate results with less space complexity.
We analyze the optimization landscape of α-loss in logistic models.
problem Optimization landscape of α-loss in logistic models.
method Tools from strictly-locally-quasi-convex functions and geometric techniques.
result Evolution of optimization landscape with respect to α.
New findings suggest adversarial training does not flatten loss landscapes, challenging current intuition.
problem Understanding and improving generalization in deep learning.
method Loss surface visualization with filter normalization technique.
result Adversarial training does not result in flatter loss landscapes, challenging current intuition.
Study reveals sharp characterisation of local minima in neural network loss landscapes.
problem Characterizing local minima in high-dimensional two-layer ReLU neural networks.
method Exact low-dimensional representation of local minima using summary statistics and link with one-pass SGD dynamics.
result Local minima in overparameterized neural networks form discrete families with varying stability and reachability.
Experimental fractal landscape dynamics observed in emulsions.
problem Understanding anomalous motions in soft glassy materials.
method Quantitative analysis of oil droplet trajectories in dense emulsions.
result Experimental fractal geometry matches computational model of soft glassy dynamics.
Study landscape of non-convex empirical risk with degenerate population risk.
problem Degenerate non-convex population risk in machine learning problems.
method Analyze population risk first, then connect to empirical risk landscape.
result Established correspondence between empirical and population risk critical points.
SGD vs quasi-Newton optimization in neural networks: different landscapes, different generalizability.
problem Understanding neural network optimization and generalizability.
method Comparison of stochastic gradient descent (SGD) and quasi-Newton optimization methods using computational tools.
result SGD solutions are separated by lower barriers than quasi-Newton solutions, but quasi-Newton solutions are deeper and more isolated.
Study visualizes actor-critic loss landscapes for inventory optimization.
problem Difficulties in solving multi-store dynamic inventory control problems.
method Low-dimensional visualizations of actor loss function.
result Loss landscapes favor optimal policies in reinforcement learning.
We study rough high-dimensional landscapes in which an increasingly stronger preference for a given configuration emerges. Such energy landscapes arise in glass physics and inference. In particular we focus on random Gaussian functions, and on the spiked-tensor model and generalizations. We thoroughly analyze the stati…
Analyzes adversarial training's impact on loss landscape, proposing PAS to improve model performance.
problem Challenges in optimizing models under adversarial training due to loss landscape properties.
method Analytical studies of adversarial loss functions, numerical analyses, PAS strategy.
result Adversarial training impairs optimization, but PAS strategy improves model performance.
LSAM optimizes deep learning training with improved efficiency.
problem Inefficiency in distributed large-batch training with Sharpness-Aware Minimization (SAM).
method Integrates SAM's adversarial steps with an asynchronous distributed sampling strategy.
result Higher final accuracy compared to data-parallel SAM.
Paper characterizes optimization landscape of Tucker decomposition.
problem Finding exact Tucker decomposition is a nonconvex optimization problem.
method Characterized the optimization landscape and provided a local search algorithm.
result All local minima are globally optimal if tensor has an exact Tucker decomposition.
Researchers improve NCE by addressing its flat loss landscape issues.
problem NCE's poor performance due to an ill-behaved loss landscape.
method Introduced eNCE with an exponential loss and normalized gradient descent.
result Proven that landscape issues arise from inappropriate noise distribution.
Embedding principle explains loss landscape of deep neural networks.
problem Understanding the structure of loss landscapes in deep neural networks.
method Proposed an embedding principle that critical points of narrower DNNs can be embedded to critical points of wider DNNs.
result Wide DNNs are often attracted by highly-degenerate critical points embedded from narrower DNNs.
The study tests inferences about neural network optimization from linear interpolation of loss landscapes.
problem Understanding the difficulty of neural network optimization problems.
method Linear interpolation of neural network loss landscapes, systematic evaluation of various factors.
result Linear interpolation does not correlate with model performance, challenging prior intuition.
The paper analyzes phase retrieval under limited samples, ensuring a benign local landscape for convergence.
problem Ensuring a benign local landscape for phase retrieval under limited samples.
method Fine-grained analysis of local landscape properties under the regime of limited samples.
result Gradient descent can converge to an od(1)-loss solution exponentially fast under certain conditions. Deep ReLU networks with extra parameters have mostly good loss landscapes.
problem Finding good local minima in the loss landscape of deep neural networks.
method Analyzing shallow and deep ReLU networks with extra parameters on a generic dataset.
result Most activation patterns correspond to regions with no bad local minima.
PHINN: A generative model for rare-event time series using persistent homology
problem Generating rare events in time series
method Flow-matching framework with dynamic Betti curves and persistence landscape loss
result Outperforms statistical and diffusion baselines in topological fidelity and tail coverage
We investigate the structure of the profit landscape obtained from the most basic, fluctuation based, trading strategy applied for the daily stock price data. The strategy is parameterized by only two variables, p and q. Stocks are sold and bought if the log return is bigger than p and less than -q, respectively. Repet…
Removes all bad local minima without adding extra units.
problem Neural network loss landscapes have bad local minima.
method Converts all bad local minima to non-local minima at infinity using auxiliary parameters.
result Removes all bad local minima from any loss landscape.
A new method uses surrogate models to speed up landscape evolution model inference.
problem Complex and computationally expensive landscape evolution models.
method Surrogate-assisted parallel tempering for Bayesian inversion.
result Significant reduction in computational cost with preserved solution quality.
This work shows how to efficiently simulate parts of quantum landscapes using classical computers.
problem Identifying where quantum computers are advantageous and offloading computations.
method Developed a quantum-enhanced classical algorithm to simulate sub-regions of quantum landscapes.
result It is possible to generate a classical surrogate of a sub-region of a quantum landscape.
This work justifies neural collapse under MSE loss and analyzes the optimization landscape.
problem Understanding neural collapse in deep neural networks under MSE loss.
method Global landscape analysis of vanilla nonconvex MSE loss.
result The only global minimizers are neural collapse solutions.
Geodesics connect model modes in neural network loss landscapes.
problem Connecting modes in neural network loss landscapes.
method Reframed mode connectivity in Information Geometry, hypothesized geodesics as mode-connecting paths, proposed algorithm to approximate geodesics.
result Geodesics achieve mode connectivity in neural networks.
Study on negative eigenvalues in deep neural networks' loss landscapes.
problem Understanding the non-convex nature of deep neural networks' loss functions.
method Examined the Hessian matrix's eigendecompositions to analyze negative eigenvalues.
result Negative eigenvalues are crucial for understanding the loss landscape of deep networks.
Model explains surprising properties of neural network training landscapes.
problem Understanding the geometry of neural network loss landscapes.
method Developed a simple theoretical model of gradients and Hessians.
result Unified model accounts for 4 surprising properties of neural loss landscapes.