Extends batch active learning to non-differentiable models.
problem Efficiently training machine learning models on large, initially unlabelled datasets.
method Black-box batch active learning for regression tasks that relies solely on model predictions.
result Achieves strong performance on regression datasets compared to white-box approaches for deep learning models.
New algorithms improve inference in non-differentiable models.
problem Inference and learning in latent variable models with non-differentiable densities.
method Proximal interacting particle Langevin algorithms (PIPLA).
result Nonasymptotic bounds and effectiveness demonstrated in various models.
This paper extends geometric study of neural networks to non-differentiable layers and random walks.
problem Understanding the geometric properties of neural networks, especially those with non-differentiable activation functions.
method Singular Riemannian geometry approach to convolutional, residual, and recursive neural networks.
result Illustrated geometric findings with numerical experiments on image classification and thermodynamic problems.
We present a comprehensive study of multilayer neural networks with binary activation, relying on the PAC-Bayesian theory. Our contributions are twofold: (i) we develop an end-to-end framework to train a binary activated deep neural network, (ii) we provide nonvacuous PAC-Bayesian generalization bounds for binary activ…
Study shows AD for neural nets with machine-representable numbers can be incorrect.
problem Correctness of AD for neural nets with machine-representable numbers.
method Analyzed two sets of parameters: incorrect and non-differentiable. Proved bounds and conditions for AD correctness.
result AD can be incorrect for machine-representable numbers, but provides a Clarke subderivative on non-differentiable set.
The study proves a quantitative functional CLT for neural networks with smooth activation functions.
problem Understanding the convergence rates of neural networks with different activation functions.
method Functional versions of the Stein-Malliavin approach and a quantitative functional central limit theorem.
result Rates of convergence depend on the smoothness of the activation function, ranging from logarithmic to sqrt(n).
The paper improves ALO for ℓ1-regularized models.
problem Estimating out-of-sample error for ℓ1-regularized models. method Developed a novel theory for ℓ1-regularized problems, bounding ALO error. result For ℓ1-regularized problems, ALO error goes to zero as p goes to infinity. New method trains neural networks with threshold activation functions efficiently.
problem Training neural networks with threshold activation functions is challenging due to zero gradients.
method We study weight decay regularized training problems of deep neural networks with threshold activations, showing they can be formulated as convex optimization problems.
result Regularized deep threshold network training problems can be formulated as standard convex optimization problems, paralleling the LASSO method.
We propose a new optimization method for training feed-forward neural networks. By rewriting the activation function as an equivalent proximal operator, we approximate a feed-forward neural network by adding the proximal operators to the objective function as penalties, hence we call the lifted proximal operator machin…
Paper proves autodiff systems are correct for non-differentiable functions.
problem Correctness of autodiff systems for non-differentiable functions in deep learning.
method Investigation of PAP functions and introduction of intensional derivatives.
result Intensional derivatives always exist and coincide with standard derivatives for almost all inputs.
A new method for discrete data normalizing flows using latent transformations.
problem Challenges in parameterizing bijective transformations for discrete data.
method Predict a distribution over latent transformations to make the marginal likelihood differentiable.
result Discrete-data normalizing flows can be trained using gradient-based learning with unbiased score function estimation.
Study particle dynamics in non-differentiable fractal spaces.
problem Understanding motion in non-smooth, probabilistic geometries.
method Use fiber bundle theory to characterize multivalued geodesic trajectories.
result Developed a hybrid theory combining surface and stochastic process theories.
Low bit-width weights and activations are an effective way of combating the increasing need for both memory and compute power of Deep Neural Networks. In this work, we present a probabilistic training method for Neural Network with both binary weights and activations, called BLRNet. By embracing stochasticity during tr…
Hamiltonian Monte Carlo on ReLU networks is inefficient due to large local error.
problem Inefficiency of Hamiltonian Monte Carlo on ReLU neural networks.
method Analysis of Hamiltonian Monte Carlo with leapfrog integrator for Bayesian neural network inference.
result Leapfrog HMC for ReLU networks has a large local error rate of Ω(ε), leading to inefficiency. We generalize stochastic smoothing for gradient estimation of non-differentiable functions.
problem Gradient estimation for non-differentiable functions.
method Developed a general framework for relaxation and gradient estimation of non-differentiable black-box functions using stochastic smoothing with reduced assumptions.
result Empirically validated the effectiveness of variance reduction strategies for various non-differentiable tasks.
Paper proposes HTAF for stable training of binary neural networks.
problem Challenges in training binary neural networks with gradient-based optimization.
method HTAF is a smooth approximation to the Heaviside function that enables stable training.
result HTAF enables stable training of various binary neural networks with gradient-based optimization.
The study connects Hilbert entropy to non-differentiability points of limit sets in flag spaces.
problem Understanding non-differentiability points in limit sets of convex projective structures.
method Introduces hyperplane conicality for θ-Anosov representations and uses it to prove properties of boundary maps. result Hilbert entropy is linked to the Hausdorff dimension of non-differentiability points in flag spaces.
Reintroduces straight-through estimators for binary neural networks.
problem Training neural networks with binary weights and activations is challenging due to gradient issues and discrete weight optimization.
method Derives ST methods as estimators in the SBN model, analyzes properties and estimation accuracy, explains latent weights and mirror descent method.
result Reintroduces ST methods as sound approximations and provides clearer application and improvements.
Study reveals properties of local minima in ReLU networks.
problem Understanding the loss landscape of neural networks.
method Theoretical analysis of one-hidden-layer ReLU networks.
result All differentiable local minima are global within certain regions.
We present a new algorithm for stochastic variational inference that targets at models with non-differentiable densities. One of the key challenges in stochastic variational inference is to come up with a low-variance estimator of the gradient of a variational objective. We tackle the challenge by generalizing the repa…
SANE improves exploration of noisy, multimodal functions by finding multiple optima.
problem Finding multiple optima in noisy, non-differentiable functions.
method Strategic Autonomous Non-Smooth Exploration (SANE) with a cost-driven acquisition function and human knowledge gate.
result SANE outperforms classical Bayesian optimization in discovering multiple optima.
Unified approach for sampling non-differentiable and heavy-tailed targets.
problem Sampling non-differentiable and heavy-tailed distributions using Langevin algorithms.
method Anchored Langevin dynamics, which modifies the Langevin diffusion with a smooth reference potential and multiplicative scaling.
result Non-asymptotic guarantees in the 2-Wasserstein distance to the target distribution.
Study improves understanding of non-differentiable penalties in high-dimensional settings.
problem Theoretical understanding of non-differentiable penalties like generalized LASSO and nuclear norm in high-dimensional settings.
method Proportional high-dimensional regime analysis with finite sample upper bounds on expected squared error.
result LO provides accurate estimation of out-of-sample risk in high-dimensional settings.
Study identifies conditions for proxy adjustment in confounded binary treatment outcomes.
problem Average causal effect estimation with a non-differentially mismeasured binary confounder.
method Identifies conditions for proxy adjustment in the presence of a non-differentially mismeasured binary confounder.
result Adjusting for a non-differentially mismeasured binary proxy can improve estimation of the average causal effect.
Differentiable pipeline replaces non-differentiable CAE components for shape optimization.
problem Gradient-based optimization is limited by non-differentiable components in CAE workflows.
method Surrogate models replace non-differentiable pipeline components, enabling gradient-based optimization.
result Gradient-based shape optimization possible without differentiable solvers.
This paper is an attempt at understanding the quantum-like dynamics of financial markets in terms of non-differentiable price-time continuum having fractal properties. The main steps of this development are the statistical scaling, the non-differentiability hypothesis, and the equations of motion entailed by this hypot…
We show that an analogue of the Ball-Box Theorem for step 2, completely non-integrable bundles from smooth sub-Riemannian geometry hold true for a class of non-differentiable tangent subbundles that satisfy a geometric condition. In the final section of the paper we give examples of such bundles and an application to d…
We present DANTE, a novel method for training neural networks using the alternating minimization principle. DANTE provides an alternate perspective to traditional gradient-based backpropagation techniques commonly used to train deep networks. It utilizes an adaptation of quasi-convexity to cast training a neural networ…
Paper generalizes Hardy-Rogers maps for market equilibrium analysis in duopoly markets.
problem Existence and uniqueness of market equilibrium in duopoly markets with non-differentiable, nonlinear response functions.
method Coupled fixed points approach for generalized Hardy-Rogers maps.
result Enriched understanding of market equilibrium in duopoly markets with non-differentiable response functions.
Proposes a method for inference in high-dimensional classification with non-differentiable surrogate losses.
problem Lack of inference procedures for identifying driving factors in high-dimensional classification with non-differentiable surrogate losses.
method Kernel-smoothed decorrelated score and cross-fitted version for hypothesis tests and interval estimators.
result Valid and superior inference methods for high-dimensional classification with non-differentiable surrogate losses.
In this work we show that Evolution Strategies (ES) are a viable method for learning non-differentiable parameters of large supervised models. ES are black-box optimization algorithms that estimate distributions of model parameters; however they have only been used for relatively small problems so far. We show that it …
We show that many machine learning goals, such as improved fairness metrics, can be expressed as constraints on the model's predictions, which we call rate constraints. We study the problem of training non-convex models subject to these rate constraints (or any non-convex and non-differentiable constraints). In the non…
We explore neural painters, a generative model for brushstrokes learned from a real non-differentiable and non-deterministic painting program. We show that when training an agent to "paint" images using brushstrokes, using a differentiable neural painter leads to much faster convergence. We propose a method for encoura…
Complex computer simulators are increasingly used across fields of science as generative models tying parameters of an underlying theory to experimental observations. Inference in this setup is often difficult, as simulators rarely admit a tractable density or likelihood function. We introduce Adversarial Variational O…
We study a nonparametric contextual bandit problem where the expected reward functions belong to a Hölder class with smoothness parameter β. We show how this interpolates between two extremes that were previously studied in isolation: non-differentiable bandits (β≤1), where rate-optimal regret is achieved by run…
Paper optimizes material microstructures with limited data using probabilistic methods.
problem Optimizing material properties with uncertain process-structure-property links.
method Flexible probabilistic formulation, data-driven surrogate, active learning.
result Significant improvement in accuracy with small training data.
Several tasks in machine learning are evaluated using non-differentiable metrics such as mean average precision or Spearman correlation. However, their non-differentiability prevents from using them as objective functions in a learning framework. Surrogate and relaxation methods exist but tend to be specific to a given…
Compared with artificial neural networks (ANNs), spiking neural networks (SNNs) are promising to explore the brain-like behaviors since the spikes could encode more spatio-temporal information. Although pre-training from ANN or direct training based on backpropagation (BP) makes the supervised training of SNNs possible…
Study calculates slope gaps on polygon surfaces, finding non-unimodal distributions.
problem Understanding the distribution of slope gaps on polygon surfaces.
method Explicit computation of slope gap distributions for 2n-gons, providing bounds on non-differentiability points.
result Slope gap distributions are not always unimodal, answering a question by Athreya.
CheXpert++ improves CheXpert's accuracy and usability for medical radiology reports.
problem Infeasibility of obtaining ground truth labels for medical data.
method BERT-based approximation of CheXpert, addressing speed, differentiability, and probabilistic output.
result Achieves 99.81% parity with CheXpert, significantly faster, differentiable, and probabilistic.
Bayesian optimization uses triangulation candidates for better performance.
problem Non-convex and multi-modal optimization challenges in Bayesian optimization.
method Proposes using Delaunay triangulation candidates for discrete search over continuous optimization.
result Triangulation candidates outperform numerically optimized and random alternatives.
This paper addresses the scalability challenge of architecture search by formulating the task in a differentiable manner. Unlike conventional approaches of applying evolution or reinforcement learning over a discrete and non-differentiable search space, our method is based on the continuous relaxation of the architectu…
Modern neural network training relies on piece-wise (sub-)differentiable functions in order to use backpropagation to update model parameters. In this work, we introduce a novel method to allow simple non-differentiable functions at intermediary layers of deep neural networks. We do so by training with a differentiable…
Unified framework for lifted training and inversion of neural networks.
problem Challenges in gradient-based training of deep neural networks.
method Unified framework encapsulating various lifted training strategies.
result Unified framework improves training landscape and stability.
New methods for calculating credit valuation adjustment with reduced noise and faster computation.
problem High statistical noise in computing sensitivities of CVA due to non-differentiable default intensities.
method Ad hoc analytical estimators to overcome non-differentiability and finite differences.
result Low statistical noise and fast computation of sensitivities to market quotes.
New method fine-tunes discrete diffusion models for RLHF tasks.
problem Fine-tuning discrete diffusion models with policy gradient methods is challenging.
method Proposed SEPO algorithm for efficient fine-tuning over non-differentiable rewards.
result Numerical experiments show scalability and efficiency of SEPO.
We propose an algorithm to calculate the exact solution for utility optimization problems on finite state spaces under a class of non-differentiable preferences. We prove that optimal strategies must lie on a discrete grid in the plane, and this allows us to reduce the dimension of the problem and define a very efficie…
This paper introduces a differentiable, scalable quantization method for neural networks.
problem Previous quantization methods lacked differentiability and scalability.
method The approach is differentiable and scalable, using bit-shifting and logarithmic quantization.
result The method achieves comparable accuracy to state-of-the-art approaches with less training time and lower inference cost.