Develops wavelet-based neural network approximation theory.
problem Analyzing neural network approximation capabilities over various activation functions.
method Wavelet frame theory on spaces of homogeneous type, sufficient conditions for approximation, error estimates.
result Derives sufficient conditions for neural networks to approximate any functions in a given space, including non-smooth activations.
Characterizes neural kernel and NNGP for various activations.
problem Understanding neural kernels and NNGP for non-RELU activations.
method Characterization of RKHS for various activation functions.
result Broad class of non-infinitely smooth activations generate equivalent RKHSs at different depths.
Smooth activations enable optimal error rates in neural networks for Sobolev function classes.
problem Achieving optimal approximation and estimation error rates for neural networks in Sobolev function classes.
method Study of neural networks with smooth activations, proving optimal rates via approximation and statistical properties.
result Constant-depth networks with smooth activations achieve optimal rates of approximation and estimation, demonstrating smoothness adaptivity.
Researchers relax the CVF's smoothness requirement to create more flexible flow models.
problem Challenges in constructing flexible density models due to the CVF's smoothness requirement.
method Introduce L \mathcal{L} L -diffeomorphisms as generalized transformations that may violate smoothness on zero Lebesgue-measure sets. result The relaxation allows for the use of non-smooth activation functions like ReLU in residual flows.
We theoretically discuss why deep neural networks (DNNs) performs better than other models in some cases by investigating statistical properties of DNNs for non-smooth functions. While DNNs have empirically shown higher performance than other standard methods, understanding its mechanism is still a challenging problem.…
Robots can rapidly acquire new skills from demonstrations. However, during generalisation of skills or transitioning across fundamentally different skills, it is unclear whether the robot has the necessary knowledge to perform the task. Failing to detect missing information often leads to abrupt movements or to collisi…
Recent research connects Hörmander's old work to modern boundary Laplacian analysis.
problem How close is the Dirichlet-to-Neumann map to the boundary Laplacian?
method Investigates techniques from Hörmander's 1950s manuscript to solve modern boundary Laplacian problems.
result Obtained results for DtN maps on non-smooth boundaries, Helmholtz equation, and differential forms.
Deep learning transforms data geometrically, akin to Ricci flow, improving classification accuracy.
problem Understanding geometric transformations in non-smooth activation functions.
method Developed a computational framework to quantify geometric changes in DNNs and introduced the concept of `global Ricci network flow`.
result Global Ricci network flow correlates with DNN accuracy, independent of network architecture and data set.
Paper tackles dynamic behavior of variable topology mechanisms, presenting new transition conditions.
problem Dynamic behavior of mechanisms with changing kinematic topology.
method Presented new transition conditions for variable topology mechanisms using projected motion equations and Voronets equations.
result Results show the dynamic behavior of joint locking in 3R and 6DOF mechanisms.
MARINA-P improves non-smooth federated optimization with adaptive stepsizes.
problem Non-smooth federated optimization in machine learning applications.
method Extends EF21-P and MARINA-P to non-smooth convex setting, proving optimal convergence rate and communication complexity bounds.
result MARINA-P achieves O ( 1 / T ) O(1/\sqrt{T}) O ( 1/ T ) convergence rate and communication complexity matching classical subgradient methods. Extends curve theory to non-smooth data with finite curvature and torsion.
problem Applying classical curve theory to non-smooth data.
method Using distributional derivative measures of functions of bounded variation.
result Essentially unique non-smooth curve solution with finite total curvature and torsion.
AsylADMM improves gossip-based learning for non-smooth objectives.
problem Efficient and robust decentralized learning on edge devices.
method Asynchronous gossip algorithm for non-smooth optimization.
result AsylADMM converges faster on non-smooth problems.
Survey on preserving curvature bounds for non-smooth Ricci flow.
problem Preserving curvature bounds for non-smooth initial data in Ricci flow.
method Survey of various weak initial data and preservation of curvature bounds.
result Various curvature lower bounds preserved up to a constant for non-smooth initial data.
We introduce non-smooth symplectic forms on manifolds and describe corresponding Poisson structures on the algebra of Colombeau generalized functions. This is achieved by establishing an extension of the classical map of smooth functions to Hamiltonian vector fields to the setting of non-smooth geometry. For mildly sin…
Adam achieves optimal convergence in deep ReLU networks via novel Kakeya bounds.
problem Training deep ReLU networks using Adam in non-smooth settings.
method Stratified Morse theory and Kakeya bounds to analyze region crossings and convergence.
result First global-optimal convergence for Adam in non-smooth, non-convex ReLU landscapes.
The paper explores various stationarity concepts in non-smooth optimization.
problem Understanding stationarity in non-smooth optimization problems.
method Introduction and discussion of different stationarity concepts for non-convex non-smooth functions.
result Clarification of the relationship among different stationarity concepts and their relevance in iterative methods.
New bounds explain deterministic non-smooth deep nets without large Lipschitz constants.
problem Challenges in explaining generalization of deterministic non-smooth deep nets.
method De-randomized PAC-Bayes margin bounds for deterministic non-convex and non-smooth predictors.
result New bounds avoid large Lipschitz constants, providing generalization guarantees.
We consider the problem of finding local minimizers in non-convex and non-smooth optimization. Under the assumption of strict saddle points, positive results have been derived for first-order methods. We present the first known results for the non-smooth case, which requires different analysis and a different algorithm…
Smoothness analysis of adversarial training reveals L ∞ L_\infty L ∞ constraints cause more non-smoothness.
problem Non-smoothness of adversarial training loss function.
method Analyzed the smoothness of adversarial training loss function using optimal attacks for model parameters.
result The L ∞ L_\infty L ∞ constraint causes more non-smoothness than L 2 L_2 L 2 constraint. It is well-known that overparametrized neural networks trained using gradient-based methods quickly achieve small training error with appropriate hyperparameter settings. Recent papers have proved this statement theoretically for highly overparametrized networks under reasonable assumptions. These results either assume…
In the framework of Lorentzian warped products, we study the Friedmann-Robertson-Walker cosmological model to investigate non-smooth curvatures associated with multiple discontinuities involved in the evolution of the universe. In particular we analyze non-smooth features of the spatially flat Friedmann-Robertson-Walke…
This work speeds up hyperparameter selection for non-smooth convex models using implicit differentiation.
problem Optimizing hyperparameters of non-smooth convex models.
method Implicit differentiation of proximal gradient and coordinate descent methods.
result Implicit differentiation can speed up hyperparameter optimization, especially for non-smooth problems.
New algorithms optimize non-smooth, non-convex objectives with improved complexity.
problem Optimizing non-smooth, non-convex stochastic objectives.
method Reduction to online learning, applying optimistic online learning techniques.
result Improved complexity for finding ( δ , ε ) (δ,ε) ( δ , ε ) -stationary points. One of the mysteries in the success of neural networks is randomly initialized first order methods like gradient descent can achieve zero training loss even though the objective function is non-convex and non-smooth. This paper demystifies this surprising phenomenon for two-layer fully connected ReLU activated neural n…
In this paper, we develop a novel {\bf ho}moto{\bf p}y {\bf s}moothing (HOPS) algorithm for solving a family of non-smooth problems that is composed of a non-smooth term with an explicit max-structure and a smooth term or a simple non-smooth term whose proximal mapping is easy to compute. The best known iteration compl…
Positive mass theorem for non-smooth metrics on flat manifolds with corners.
problem Proving a positive mass theorem for non-smooth metrics on asymptotically flat manifolds with non-compact boundary.
method Proves a positive mass theorem for metrics that are only continuous across a compact hypersurface.
result Obtains a positive mass theorem on manifolds with non-compact corners.
Paper tackles private optimization for non-smooth objectives efficiently.
problem Private stochastic convex optimization for non-smooth objectives.
method Noisy mirror descent algorithm.
result Achieves optimal rates in statistical complexity and number of queries.
We investigate a generalization of the so-called metric splitting of globally hyperbolic space-times to non-smooth Lorentzian manifolds and show the existence of this metric splitting for a class of wave-type space-times. Our approach is based on smooth approximations of non-smooth space-times by families (or sequences…
Advances smooth over-parameterization for solving non-smooth optimization problems.
problem Non-smooth optimization with structural constraints in imaging and machine learning.
method Smooth over-parameterization of non-smooth problems, using gradient descent and mirror descent.
result Gradient descent on the reformulated smooth problem converges efficiently without parameter tuning.
New methods improve convergence in non-convex non-smooth learning problems.
problem Sparse learning from high-dimensional data with non-convex, non-smooth regularizers.
method Stochastic proximal gradient methods with arbitrary sampling.
result Independent sampling improves performance over uniform sampling.
SGD avoids critical points on weakly convex functions.
problem Non-convergence of SGD to critical points on specific manifolds.
method Stochastic subgradient descent, Verdier stratification, angle condition.
result SGD converges to local minimizers on weakly convex functions.
The study analyzes convergence rates for sparse pivotal estimators in high-dimensional regression.
problem Sparse pivotal estimation in high-dimensional regression problems.
method Theoretical analysis and comparison of non-smooth + non-smooth optimization problems, including smoothing techniques.
result Minimax sup-norm convergence rates for square-root Lasso-type estimators are derived.
Adaptive data fusion boosts efficiency in multi-task optimization.
problem Multi-task non-smooth optimization in various fields.
method Adaptive data fusion approach leveraging commonalities among objectives.
result Significant improvements in sample efficiency with sharp statistical guarantees.
Algebras of generalized functions offer possibilities beyond the purely distributional approach in modelling singular quantities in non-smooth differential geometry. This article presents an introductory survey of recent developments in this field and highlights some applications in mathematical physics.
Abstracts a theorem for non-smooth maps in infinite dimensions.
problem Generalizing inverse mapping theorem for non-smooth maps.
method Introduces property A and applies it to non-smooth maps.
result Generalized inverse mapping theorems for non-smooth maps.
New SPS variant improves non-smooth optimization without small gradients.
problem Improving non-smooth optimization without small gradients.
method Safeguarded Stochastic Polyak Step Size (SPS s a f e _{safe} s a f e ) for non-smooth optimization. result Rigorous convergence guarantees for non-smooth convex optimization without strong assumptions.
Annealed Entropic Allocation improves ranking and selection by mitigating hard switching and improving finite-budget discrimination.
problem Sequential budget allocation in ranking and selection
method Annealed weighted soft-min framework
result Surrogate converges uniformly to the hard minimum, soft-min weights concentrate on active challengers, and target allocation map is continuous.
Stochastic approximation proves asymptotic normality for non-smooth problems.
problem Solving non-smooth stochastic approximation problems.
method Stochastic approximation algorithms for solving smooth equations, extended to non-smooth problems.
result Asymptotic normality and optimality in non-smooth stochastic approximation is proven.
Bayesian optimization tackles non-smooth tuning problems.
problem Optimizing black-box functions with non-smoothness and limited samples.
method Proposed a clustered Gaussian process (cGP) model for non-smooth optimization.
result Improvement of up to 90% in performance for repetitive experiments.
We provide improved convergence rates for various \emph{non-smooth} optimization problems via higher-order accelerated methods. In the case of ℓ ∞ \ell_\infty ℓ ∞ regression, we achieves an O ( ε − 4 / 5 ) O(ε^{-4/5}) O ( ε − 4/5 ) iteration complexity, breaking the O ( ε − 1 ) O(ε^{-1}) O ( ε − 1 ) barrier so far present for previous methods. We arrive at a similar rate fo…
How can local-search methods such as stochastic gradient descent (SGD) avoid bad local minima in training multi-layer neural networks? Why can they fit random labels even given non-convex and non-smooth architectures? Most existing theory only covers networks with one hidden layer, so can we go deeper? In this paper, w…
Safe-EF improves federated learning for non-smooth, constrained optimization.
problem Federated learning's communication bottlenecks with high-dimensional model updates.
method Error feedback (EF) for non-smooth convex optimization with safety constraints.
result Safe-EF matches lower complexity bounds and ensures safety constraints.
Modified perturbation method removes non-smoothness in solving Black-Scholes equations.
problem Non-smoothness in solving Black-Scholes equations.
method Variable transformations and homotopy perturbation method.
result Excellent agreement with exact solutions for Black-Scholes and multi-asset options.
The study extends curvature bounds to non-smooth spaces and proves stability of mean curvature.
problem Proving curvature bounds in non-smooth spaces.
method Extending results from smooth Riemannian manifolds to non-smooth RCD spaces.
result Stability of mean curvature bounds under uniform convergence.
Expanding FCCO to non-smooth weakly-convex problems, improving deep learning performance.
problem Addressing the limitations of current FCCO methods by tackling non-smooth weakly-convex problems.
method Developed a single-loop algorithm for non-smooth weakly-convex FCCO and extended it to tri-level problems.
result Established the complexity for finding ε-stationary points in the Moreau envelop of the objective function.
Novel method for shape optimization of non-smooth PDEs.
problem Optimizing shapes governed by non-smooth PDEs.
method Functional variational approach and sensitivity analysis.
result Necessary conditions for locally optimal shapes.
Example of non-smooth isotopy using K3 surfaces.
problem Non-smooth isotopy examples in 4-manifolds.
method Dehn twist along a 3-sphere in K3 surfaces' connected sum.
result First example of self-diffeomorphisms isotopic to identity but not smoothly.
Injectivity of X-ray transform proven for non-smooth metrics.
problem Injectivity of X-ray transform on non-smooth metrics.
method Microlocal analysis of the normal operator, establishing ellipticity and smoothing properties.
result Injectivity of X-ray transform on L 2 L^2 L 2 for metrics with finitely differentiable tensor.