The paper improves ALO for ℓ1-regularized models.
problem Estimating out-of-sample error for ℓ1-regularized models. method Developed a novel theory for ℓ1-regularized problems, bounding ALO error. result For ℓ1-regularized problems, ALO error goes to zero as p goes to infinity. The study connects Hilbert entropy to non-differentiability points of limit sets in flag spaces.
problem Understanding non-differentiability points in limit sets of convex projective structures.
method Introduces hyperplane conicality for θ-Anosov representations and uses it to prove properties of boundary maps. result Hilbert entropy is linked to the Hausdorff dimension of non-differentiability points in flag spaces.
New methods approximate LOOCV for high-dimensional, non-differentiable learning problems.
problem Finding optimal regularization parameters in high-dimensional learning problems.
method Three frameworks based on primal, dual, and proximal formulations of a convex optimization problem.
result Equivalence of three methods under smoothness conditions, validated by empirical results.
New stochastic algorithms solve DC functions and non-convex problems efficiently.
problem Solving non-convex, non-smooth, and non-differentiable functions efficiently.
method Proposed new stochastic optimization algorithms for DC functions and non-convex problems.
result First non-asymptotic convergence for non-convex optimization with general non-convex non-differentiable regularizers.
Proposes a method for inference in high-dimensional classification with non-differentiable surrogate losses.
problem Lack of inference procedures for identifying driving factors in high-dimensional classification with non-differentiable surrogate losses.
method Kernel-smoothed decorrelated score and cross-fitted version for hypothesis tests and interval estimators.
result Valid and superior inference methods for high-dimensional classification with non-differentiable surrogate losses.
Study improves understanding of non-differentiable penalties in high-dimensional settings.
problem Theoretical understanding of non-differentiable penalties like generalized LASSO and nuclear norm in high-dimensional settings.
method Proportional high-dimensional regime analysis with finite sample upper bounds on expected squared error.
result LO provides accurate estimation of out-of-sample risk in high-dimensional settings.
Deep weight factorization improves neural network training through smooth optimization of sparse penalties.
problem Challenges in applying sparse regularization in neural networks due to non-differentiability of penalties.
method Introduces deep weight factorization, decomposing weights into multiple factors for smooth optimization of L1-penalized networks. result Deep weight factorization outperforms shallow factorization and pruning methods consistently across various architectures and datasets.
Paper proves autodiff systems are correct for non-differentiable functions.
problem Correctness of autodiff systems for non-differentiable functions in deep learning.
method Investigation of PAP functions and introduction of intensional derivatives.
result Intensional derivatives always exist and coincide with standard derivatives for almost all inputs.
Extends batch active learning to non-differentiable models.
problem Efficiently training machine learning models on large, initially unlabelled datasets.
method Black-box batch active learning for regression tasks that relies solely on model predictions.
result Achieves strong performance on regression datasets compared to white-box approaches for deep learning models.
New algorithms improve inference in non-differentiable models.
problem Inference and learning in latent variable models with non-differentiable densities.
method Proximal interacting particle Langevin algorithms (PIPLA).
result Nonasymptotic bounds and effectiveness demonstrated in various models.
Study particle dynamics in non-differentiable fractal spaces.
problem Understanding motion in non-smooth, probabilistic geometries.
method Use fiber bundle theory to characterize multivalued geodesic trajectories.
result Developed a hybrid theory combining surface and stochastic process theories.
We generalize stochastic smoothing for gradient estimation of non-differentiable functions.
problem Gradient estimation for non-differentiable functions.
method Developed a general framework for relaxation and gradient estimation of non-differentiable black-box functions using stochastic smoothing with reduced assumptions.
result Empirically validated the effectiveness of variance reduction strategies for various non-differentiable tasks.
Differentiable PF via entropy-regularized OT for better inference.
problem Non-differentiability of traditional PF resampling methods.
method Entropy-regularized optimal transport for differentiable resampling.
result Convergent differentiable PF method with improved gradient estimates.
Paper develops a method to learn optimal sparsity-promoting regularizers for linear inverse problems.
problem Solving linear inverse problems with sparse solutions.
method Bilevel optimization framework to select an optimal synthesis operator B. result Established well-posedness and theoretical guarantees for the learning process.
ES for non-differentiable parameters scales to large models.
problem Learning non-differentiable parameters in large models.
method Hybrid approach combining ES for non-differentiable and gradient-based methods for differentiable parameters.
result Hybrid approach is competitive and allows training sparse models from the start.
New method solves non-convex constrained optimization problems with non-differentiable constraints.
problem Training non-convex models with non-differentiable constraints.
method Proxy-Lagrangian formulation and semi-coarse correlated equilibrium.
result Solves non-convex constrained optimization problems with theoretical guarantees.
Algorithm finds optimal investment strategies for non-differentiable preferences.
problem Optimal investment strategies under non-differentiable preferences.
method Reduces problem to a discrete grid, uses efficient method to find strategies.
result Optimal strategies lie on a discrete grid, allowing efficient computation.
We present a new algorithm for stochastic variational inference that targets at models with non-differentiable densities. One of the key challenges in stochastic variational inference is to come up with a low-variance estimator of the gradient of a variational objective. We tackle the challenge by generalizing the repa…
We introduce non-smooth symplectic forms on manifolds and describe corresponding Poisson structures on the algebra of Colombeau generalized functions. This is achieved by establishing an extension of the classical map of smooth functions to Hamiltonian vector fields to the setting of non-smooth geometry. For mildly sin…
Study shows AD for neural nets with machine-representable numbers can be incorrect.
problem Correctness of AD for neural nets with machine-representable numbers.
method Analyzed two sets of parameters: incorrect and non-differentiable. Proved bounds and conditions for AD correctness.
result AD can be incorrect for machine-representable numbers, but provides a Clarke subderivative on non-differentiable set.
Unified approach for sampling non-differentiable and heavy-tailed targets.
problem Sampling non-differentiable and heavy-tailed distributions using Langevin algorithms.
method Anchored Langevin dynamics, which modifies the Langevin diffusion with a smooth reference potential and multiplicative scaling.
result Non-asymptotic guarantees in the 2-Wasserstein distance to the target distribution.
We propose a version of least-mean-square (LMS) algorithm for sparse system identification. Our algorithm called online linearized Bregman iteration (OLBI) is derived from minimizing the cumulative prediction error squared along with an l1-l2 norm regularizer. By systematically treating the non-differentiable regulariz…
Study identifies conditions for proxy adjustment in confounded binary treatment outcomes.
problem Average causal effect estimation with a non-differentially mismeasured binary confounder.
method Identifies conditions for proxy adjustment in the presence of a non-differentially mismeasured binary confounder.
result Adjusting for a non-differentially mismeasured binary proxy can improve estimation of the average causal effect.
Develops LF-PPL for non-differentiable models with automatic boundary checks.
problem Handling non-differentiable models in probabilistic programming.
method Introduces LF-PPL with automatic boundary checks and a formalism ensuring measure zero discontinuities.
result Demonstrates efficient inference for non-differentiable models using DHMC.
Differentiable pipeline replaces non-differentiable CAE components for shape optimization.
problem Gradient-based optimization is limited by non-differentiable components in CAE workflows.
method Surrogate models replace non-differentiable pipeline components, enabling gradient-based optimization.
result Gradient-based shape optimization possible without differentiable solvers.
This paper is an attempt at understanding the quantum-like dynamics of financial markets in terms of non-differentiable price-time continuum having fractal properties. The main steps of this development are the statistical scaling, the non-differentiability hypothesis, and the equations of motion entailed by this hypot…
We show that an analogue of the Ball-Box Theorem for step 2, completely non-integrable bundles from smooth sub-Riemannian geometry hold true for a class of non-differentiable tangent subbundles that satisfy a geometric condition. In the final section of the paper we give examples of such bundles and an application to d…
Solutions to a specific problem are shown to be locally Lipschitz but not differentiable.
problem Locally Lipschitz viscosity solutions to the σk-Loewner-Nirenberg problem on annuli. method Analytical proof of regularity and non-differentiability.
result Solutions are $C^{1, rac{1}{k}}_{
m loc}$ in each of the annulus regions and have a jump in radial derivative.
Smooth Contextual Bandits bridge two previously studied extremes of non-differentiable and parametric-response bandits.
problem Nonparametric contextual bandits with Hölder smoothness.
method Developed a novel algorithm that optimally balances between non-differentiable and parametric-response bandits.
result Proved the algorithm achieves rate-optimal regret for all smoothness settings.
SoDeep learns approximations of ranking metrics for deep learning tasks.
problem Non-differentiable metrics in machine learning tasks.
method Sorting deep (SoDeep) net trained to approximate sorting of scores.
result Competitive results on Cross-modal text-image retrieval, multi-label image classification, and visual memorability ranking tasks.
Paper generalizes Hardy-Rogers maps for market equilibrium analysis in duopoly markets.
problem Existence and uniqueness of market equilibrium in duopoly markets with non-differentiable, nonlinear response functions.
method Coupled fixed points approach for generalized Hardy-Rogers maps.
result Enriched understanding of market equilibrium in duopoly markets with non-differentiable response functions.
Paper proposes a method to minimize non-differentiable loss functions.
problem Minimizing non-differentiable and non-decomposable loss functions.
method Learn smooth relaxations of true losses through surrogate neural networks, then optimize jointly with the prediction model.
result Empirical results show the efficiency of learning surrogate losses.
We apply the OSCAR (octagonal selection and clustering algorithms for regression) in recovering group-sparse matrices (two-dimensional---2D---arrays) from compressive measurements. We propose a 2D version of OSCAR (2OSCAR) consisting of the ℓ1 norm and the pair-wise ℓ∞ norm, which is convex but non-d…
Complex computer simulators are increasingly used across fields of science as generative models tying parameters of an underlying theory to experimental observations. Inference in this setup is often difficult, as simulators rarely admit a tractable density or likelihood function. We introduce Adversarial Variational O…
The study proves a quantitative functional CLT for neural networks with smooth activation functions.
problem Understanding the convergence rates of neural networks with different activation functions.
method Functional versions of the Stein-Malliavin approach and a quantitative functional central limit theorem.
result Rates of convergence depend on the smoothness of the activation function, ranging from logarithmic to sqrt(n).
Proximal boosting improves gradient boosting for non-differentiable losses.
problem Minimizing non-differentiable losses in prediction models.
method Proximal point algorithm applied to gradient boosting.
result Proximal boosting outperforms gradient boosting in convergence rate and accuracy.
Consider the following class of learning schemes: β^:=argminβ∑j=1nℓ(xj⊤β;yj)+λR(β),(1) where xi∈Rp and yi∈R denote the ith feature and response variable …
We propose a practical method for L0 norm regularization for neural networks: pruning the network during training by encouraging weights to become exactly zero. Such regularization is interesting since (1) it can greatly speed up training and inference, and (2) it can improve generalization. AIC and BIC, well-known …
Integrates estimation and optimization for uncertain parameters.
problem Optimizing with uncertain parameters whose distributions can be estimated.
method Integrated Conditional Estimation-Optimization (ICEO) framework.
result Asymptotically consistent and provides finite performance guarantees.
This paper extends geometric study of neural networks to non-differentiable layers and random walks.
problem Understanding the geometric properties of neural networks, especially those with non-differentiable activation functions.
method Singular Riemannian geometry approach to convolutional, residual, and recursive neural networks.
result Illustrated geometric findings with numerical experiments on image classification and thermodynamic problems.
The Support Vector Machine (SVM) has been used in a wide variety of classification problems. The original SVM uses the hinge loss function, which is non-differentiable and makes the problem difficult to solve in particular for regularized SVMs, such as with ℓ1-regularization. This paper considers the Huberized SV…
Study calculates slope gaps on polygon surfaces, finding non-unimodal distributions.
problem Understanding the distribution of slope gaps on polygon surfaces.
method Explicit computation of slope gap distributions for 2n-gons, providing bounds on non-differentiability points.
result Slope gap distributions are not always unimodal, answering a question by Athreya.
ALO-CV approximates leave-one-out error in proportional regime.
problem Estimating generalization error in high-dimensional settings.
method Developed new analysis for ALO-CV, showed consistency under strong convexity.
result ALO-CV approximates leave-one-out error up to negligible error.
Bayesian optimization uses triangulation candidates for better performance.
problem Non-convex and multi-modal optimization challenges in Bayesian optimization.
method Proposes using Delaunay triangulation candidates for discrete search over continuous optimization.
result Triangulation candidates outperform numerically optimized and random alternatives.
This paper addresses the scalability challenge of architecture search by formulating the task in a differentiable manner. Unlike conventional approaches of applying evolution or reinforcement learning over a discrete and non-differentiable search space, our method is based on the continuous relaxation of the architectu…
New method trains neural networks with threshold activation functions efficiently.
problem Training neural networks with threshold activation functions is challenging due to zero gradients.
method We study weight decay regularized training problems of deep neural networks with threshold activations, showing they can be formulated as convex optimization problems.
result Regularized deep threshold network training problems can be formulated as standard convex optimization problems, paralleling the LASSO method.
New machine learning method uses algorithmic complexity for non-differentiable spaces.
problem Machine learning on non-differentiable spaces.
method Introduces complexity theory in machine learning, using algorithmic complexity for regression and classification.
result More generalizable and resilient to random attacks compared to traditional methods.
New method fine-tunes discrete diffusion models for RLHF tasks.
problem Fine-tuning discrete diffusion models with policy gradient methods is challenging.
method Proposed SEPO algorithm for efficient fine-tuning over non-differentiable rewards.
result Numerical experiments show scalability and efficiency of SEPO.