Wasserstein GANs fail to approximate Wasserstein distance, leading to their success.
problem Approximating Wasserstein distance in deep generative models.
method Analysis of differences between theoretical setup and training reality.
result Wasserstein GANs' success is due to their failure to approximate Wasserstein distance.
In this paper, we propose a successive convex approximation framework for sparse optimization where the nonsmooth regularization function in the objective function is nonconvex and it can be written as the difference of two convex functions. The proposed framework is based on a nontrivial combination of the majorizatio…
Paper tackles non-convex optimization for higher moments in portfolio management.
problem Complexity of higher moments in optimization problems.
method Method of successive convex approximation.
result Solves mean-variance-skewness problem using non-convex optimization.
Recently, Petrik et al. demonstrated that L1Regularized Approximate Linear Programming (RALP) could produce value functions and policies which compared favorably to established linear value function approximation techniques like LSPI. RALP's success primarily stems from the ability to solve the feature selection and va…
A new knot selection method speeds up sparse Gaussian process approximations.
problem Efficiently selecting knots for sparse Gaussian processes.
method One-at-a-time Bayesian optimization for knot selection.
result Competitive performance with reduced computational cost.
Quantile Temporal-Difference learning proved convergent with proof.
problem Lack of theoretical understanding of QTD despite empirical success.
method Proof of convergence using stochastic approximation and non-smooth analysis.
result QTD converges to fixed points with probability 1.
MLPs can approximate any function in context, challenging the importance of in-context universality.
problem Understanding why transformers are more effective than classical models.
method Proved MLPs with trainable activation functions are universal in context.
result Transformer success is likely due to factors other than in-context universality.
Unified framework for multi-view learning with orthogonal projections.
problem Learning individual orthogonal projections for multiple views.
method Successive approximations via eigenvectors, iterative Krylov subspace method.
result Consistently competitive and often better than existing methods.
This research solves Plateau's problem for CRPC surfaces.
problem Constructing surfaces with constant ratio of principal curvatures.
method Proposed a family of surfaces containing a given minimal surface without flat points.
result Obtained a partial solution to Plateau's problem for CRPC surfaces.
Paper proposes an efficient algorithm to handle high-order portfolio moments.
problem Designing portfolios with high-order moments (skewness and kurtosis) is computationally challenging.
method Proposes a SCA algorithm framework for solving high-order portfolios efficiently.
result Demonstrates the efficiency of the proposed algorithm through numerical experiments.
Momentum Stochastic Gradient Descent (MSGD) algorithm has been widely applied to many nonconvex optimization problems in machine learning, e.g., training deep neural networks, variational Bayesian inference, and etc. Despite its empirical success, there is still a lack of theoretical understanding of convergence proper…
Residual flows are shown to approximate MMD well.
problem Lack of theoretical understanding of normalizing flows' expressiveness.
method Proved residual flows are universal approximators in MMD.
result Residual flows can approximate MMD with a bounded number of blocks.
Deep networks can approximate functions with fewer learnable parameters than previously thought.
problem High computational costs due to large number of parameters in deep neural networks.
method Theoretical design of ReLU networks with a few intrinsic parameters and numerical experiments.
result ReLU networks with a small number of intrinsic parameters can achieve good approximations of functions.
The paper explores how market trade values and volumes affect price and return statistics.
problem Understanding the statistical properties of market trade, price, and return.
method Introduces secondary averaging procedure to describe statistical moments of market trades, price, and return.
result Predictions of market-based probabilities of price and return are limited by Gaussian distributions.
Established techniques for simulation and prediction with Gaussian process (GP) dynamics often implicitly make use of an independence assumption on successive function evaluations of the dynamics model. This can result in significant error and underestimation of the prediction uncertainty, potentially leading to failur…
The successive projection algorithm (SPA) can quickly solve a nonnegative matrix factorization problem under a separability assumption. Even if noise is added to the problem, SPA is robust as long as the perturbations caused by the noise are small. In particular, robustness against noise should be high when handling th…
Many introductory courses in quantum mechanics include Feynman's time-slicing definition of the path integral, with a complete derivation of the propagator in the simplest of cases. However, attempts to generalize this, for instance to non-quadratic potentials, encounter formidable analytic issues in showing the succes…
New method improves training-free guidance for diffusion models, achieving state-of-the-art results.
problem Accurate, training-free guidance for conditional generation in diffusion models.
method Sequential Monte Carlo (SMC) framework with Multi-Level Monte Carlo (MLMC) variance reduction.
result Achieves state-of-the-art results on CIFAR-10 and ImageNet datasets with significant cost reduction.
Deep ReLU networks can approximate matrix-vector products with error bounds.
problem Can deep ReLU networks accurately approximate matrix-vector products?
method Derived error bounds in Lebesgue and Sobolev norms for deep ReLU FNNs.
result Developed deep approximation theory with successful applications.
CVNNs improve performance in tasks with complex-valued inputs.
problem Improving performance in tasks with complex-valued inputs.
method Analyze the approximation properties of complex-valued neural networks (CVNNs).
result Quantitative approximation bounds for CVNNs, showing error scales as m−k/(2n). We study the effects of approximate inference on the performance of Thompson sampling in the k-armed bandit problems. Thompson sampling is a successful algorithm for online decision-making but requires posterior inference, which often must be approximated in practice. We show that even small constant inference error …
Study analyzes deep learning's performance on variable exponent Besov space, highlighting adaptivity benefits.
problem Estimation error analysis of deep learning in variable exponent Besov space.
method Analysis of general approximation error and estimation errors of deep learning.
result Adaptivity of deep learning leads to significant improvement in estimation error, especially in high-dimensional spaces.
Bayesian approach learns invariances from data alone, but last layer approximation is not always sufficient.
problem Learning invariances in neural networks using only training data.
method Bayesian marginal likelihood for last layer, custom optimisation routine, new lower bound.
result Partial success on standard benchmarks and medical imaging dataset, failure on CIFAR10.
Meta-algorithm selects optimal model complexity in RL with function approximation.
problem Selecting the right model complexity for reinforcement learning with function approximation.
method Meta-algorithm that successively rejects complex models using a statistical test.
result Adapts to optimal complexity with improved regret bounds compared to best candidate.
This work analyzes nonexpansive stochastic approximations with Markovian noise, proving convergence in reinforcement learning.
problem Applying stochastic approximation to reinforcement learning settings with nonexpansive operators.
method Investigates nonexpansive stochastic approximations with Markovian noise, providing asymptotic and finite sample analysis.
result First-time proof of convergence for classical tabular average reward temporal difference learning.
A new machine learning methodology, called successive subspace learning (SSL), is introduced in this work. SSL contains four key ingredients: 1) successive near-to-far neighborhood expansion; 2) unsupervised dimension reduction via subspace approximation; 3) supervised dimension reduction via label-assisted regression …
New model generates images by reversing heat equation, revealing disentanglement.
problem Image generation without considering image structure.
method Stochastically reverses the heat equation to generate images, using variational approximation.
result Emergent disentanglement of overall colour and shape in images.
New methods stabilize Q-learning with linear approximations.
problem Stabilizing Q-learning with linear function approximation. method Target network and truncation.
result Provably stable Q-learning with linear function approximation. Sequential Monte Carlo (SMC) methods comprise one of the most successful approaches to approximate Bayesian filtering. However, SMC without good proposal distributions struggle in high dimensions. We propose nested sequential Monte Carlo (NSMC), a methodology that generalises the SMC framework by requiring only approxi…
This paper proposes a new family of algorithms for training neural networks (NNs). These are based on recent developments in the field of non-convex optimization, going under the general name of successive convex approximation (SCA) techniques. The basic idea is to iteratively replace the original (non-convex, highly d…
Paper establishes convergence rates and concentration bounds for stochastic approximation and reinforcement learning with Markovian noise.
problem Analyzing convergence rates and concentration bounds for stochastic approximation and reinforcement learning with Markovian noise.
method Novel discretization of the mean ODE of stochastic approximation algorithms using intervals with diminishing length.
result First almost sure convergence rate and maximal concentration bound with exponential tails for contractive stochastic approximation algorithms with Markovian noise.
This work investigates how multi-round reasoning improves LLM performance.
problem Improving problem-solving abilities in complex tasks with LLMs.
method Investigates approximation, learnability, and generalization properties of multi-round auto-regressive models.
result Transformers with finite context windows are universal approximators for Turing-computable functions and can approximate any Turing-computable sequence-to-sequence function through multi-round reasoning.
ALBU improves LDA performance on small datasets.
problem Limited data impairs variational Bayes in LDA.
method Approximate Loopy Belief message passing algorithm.
result ALBU learns latent distributions more accurately than VB, especially for smaller data sets.
New method speeds up kernel-based machine learning for force field reconstruction.
problem Scalability issues in kernel-based machine learning for force field reconstruction.
method Nyström-type methods to construct preconditioners based on low-rank approximations of the kernel matrix.
result Effective preconditioners lead to super-linear convergence in kernel-based machine learning.
Gaussian processes have been successful in both supervised and unsupervised machine learning tasks, but their computational complexity has constrained practical applications. We introduce a new approximation for large-scale Gaussian processes, the Gaussian Process Random Field (GPRF), in which local GPs are coupled via…
Study analyzes error in ReLU networks with local connections.
problem Improving neural network performance and understanding approximation errors.
method Analyzed approximation error of ReLU networks with local connections.
result Error estimate depends on depth and width of hidden layers.
Gradient ascent method successfully removes specific data points from neural networks without retraining.
problem Addressing privacy and ethical concerns by removing specific data points from trained models.
method Gradient ascent approach to unlearning, leveraging the implicit bias of gradient descent towards margin maximization conditions.
result Gradient ascent method can successfully unlearn specific data points from two-layer ReLU neural networks without retraining.
This paper describes a hierarchical learning strategy for generating sparse representations of multivariate datasets. The hierarchy arises from approximation spaces considered at successively finer scales. A detailed analysis of stability, convergence and behavior of error functionals associated with the approximations…
A new method improves ICA performance by approximating MDI.
problem Improving F astICA's performance with nonlinear functions.
method Second-order approximation of MDI for joint maximization.
result Efficiency validated through experiments compared to other ICA algorithms.
This paper proposes low-complexity algorithms for finding approximate second-order stationary points (SOSPs) of problems with smooth non-convex objective and linear constraints. While finding (approximate) SOSPs is computationally intractable, we first show that generic instances of the problem can be solved efficientl…
In this paper we present qualitative and quantitative comparison of various analytical and numerical approximation methods for calculating a position of the early exercise boundary of the American put option paying zero dividends. First we analyze their asymptotic behavior close to expiration. In the second part of the…
We consider a class of nonconvex nonsmooth optimization problems whose objective is the sum of a smooth function and a finite number of nonnegative proper closed possibly nonsmooth functions (whose proximal mappings are easy to compute), some of which are further composed with linear maps. This kind of problems arises …
Entropy-regularized NPG converges linearly with linear function approximation.
problem Analyzing convergence of entropy-regularized NPG with function approximation.
method Established finite-time convergence analyses with entropy regularization and linear function approximation.
result Entropy-regularized NPG achieves linear convergence up to a function approximation error.
Transformers handle infinite dimensional inputs effectively by feature extraction and dynamic feature selection.
problem Understanding the approximation and estimation ability of Transformers with infinite dimensional inputs.
method Anisotropic smoothness analysis and feature extraction properties of Transformers.
result Transformers avoid the curse of dimensionality and dynamically select important features.
This work improves sample efficiency in neural function approximation for reinforcement learning.
problem Improving sample efficiency in reinforcement learning with neural function approximation.
method Study of function approximation with two-layer neural networks (ReLU and polynomial activations) under generative and realizability models.
result Significant improvement in sample complexity compared to linear methods.
This work explains GANs as Bayesian neural networks with partial stochasticity.
problem Challenges in optimizing GANs and understanding their limitations.
method Interpreting GANs as Bayesian neural networks with partial stochasticity, establishing conditions, and proposing strategies to smooth the loss landscape and find solutions with minimum description length.
result Proposed strategies lead to performance improvements and deeper understanding of GANs.
Integrates ESG data into Black-Litterman for portfolio optimization.
problem Optimizing portfolios with ESG considerations.
method Black-Litterman framework with Stein shrinkage for ESG bias, multivariate affine normal-inverse Gaussian model, CVaR risk measure, daily reallocation.
result Successful portfolio optimization with returns of 40-45% annually.
Q-learning with neural network function approximation (neural Q-learning for short) is among the most prevalent deep reinforcement learning algorithms. Despite its empirical success, the non-asymptotic convergence rate of neural Q-learning remains virtually unknown. In this paper, we present a finite-time analysis of a…