Evolutionary algorithms improve neural network performance by discovering better activation functions.
problem The choice of activation function affects neural network performance, but ReLU remains dominant.
method Defined a tree-based search space of candidate activation functions and used evolutionary algorithms (mutation, crossover, exhaustive search) to explore and discover better functions.
result Replacing ReLU with evolved activation functions statistically significantly increases network accuracy.
Paper introduces a new curriculum generation method for reinforcement learning.
problem Improving reinforcement learning performance and speed through curriculum learning.
method The paper proposes a novel curriculum generation paradigm based on progression and mapping functions.
result Empirical results show the new approach outperforms state-of-the-art algorithms.
Introduces new performance measures using scaled utility functions.
problem Performance measurement in financial contexts.
method Certainty equivalents defined via scaled utility functions, well-posed portfolio optimization problem under generic conditions.
result Link between portfolio dynamics, benchmark process, and utility function choice in the long-run setting.
An appropriate choice of the activation function (like ReLU, sigmoid or swish) plays an important role in the performance of (deep) multilayer perceptrons (MLP) for classification and regression learning. Prototype-based classification learning methods like (generalized) learning vector quantization (GLVQ) are powerful…
Develops robust MDPs for unknown disturbances with performance guarantees.
problem Unknown disturbance distribution in MDPs.
method Empirical distribution, sublevel set of distance function, weak convergence, concentration inequality.
result Robust optimal value function converges to true optimal value function with increasing sample sizes.
The paper explores conditions for predicting optimization performance.
problem Lack of formal theoretical guarantees linking prediction and optimization performance.
method Exploring conditions for asymptotic convergence and exact quantification of optimization performance.
result Explicit theoretical relationship between prediction and optimization performance.
Deep neural networks perform poorly on smooth functions despite strong approximation theory.
problem The gap between deep learning theory and practical performance on smooth functions.
method Computational framework to study DNN performance, comparing against best-in-class methods.
result There is a crucial gap between DNN approximation theory and practical performance, but it can be closed.
Boosts Q-learning by using value function bounds.
problem Efficiently solving new tasks using past experience.
method Derives double-sided bounds on optimal value function and uses them to update Q-function.
result Boosted training performance through alternative Q-function update method.
Study finds simple model-agreement scores perform well in various error estimation scenarios.
problem Evaluating model performance on unseen distributions using disparate scoring functions.
method Rigorously studied popular scoring functions (confidence, local manifold smoothness, model agreement) independently of mechanism choice.
result Simple model-agreement scores outperform confidence- and smoothness-based scores in realistic settings with compromised training data.
Optimizes resource allocation for virtualized network functions based on performance profiles.
problem Mapping SLA performance requirements to dynamic virtualized infrastructure resources.
method Profile-based resource allocation using VNF performance datasets and machine learning models.
result A method to predict and recommend optimal resource allocation for network services.
New approach to handle ranking function variation in zero-shot NAS.
problem Variation in ranking function outputs due to randomness.
method Viewing ranking function output as a random variable and constructing a stochastic ordering.
result Stochastic ordering boosts performance in neural architecture search.
Paper optimizes DC pension fund management with VaR and relative performance constraints.
problem Optimizing DC pension fund performance under VaR and relative performance constraints.
method Introduced an auxiliary process to transform the problem into a self-financing problem, combined linearization, Lagrange dual, martingale, and concavification methods.
result Explicit investment strategies obtained for certain penalty and reward functions.
Automatically discovers effective activation functions for deep learning.
problem Inconsistent performance of novel activation functions in deep learning networks.
method Evolutionary search for general form, gradient descent for parameters.
result Significant performance improvements over ReLU and other functions.
Optimized nonlinearities enhance generalization in random feature models.
problem Improving generalization performance in random feature models.
method Analyzed and defined optimal nonlinearities from Gaussian model parameters.
result Optimized nonlinearities achieve better generalization performance than ReLU.
Survey of trainable activation functions in neural networks.
problem Improving neural network performance through trainable activation functions.
method Taxonomy and comparison of recent and past models of trainable activation functions.
result Many trainable activation functions are equivalent to adding neuron layers with fixed activation functions and simple constraints.
Study proposes a differentiable surrogate loss function for optimizing Fβ score in binary classification with imbalanced data.
problem Non-differentiability of Fβ score makes it unsuitable for optimization by gradient-based learning. method Investigated relationship between Fβ score and loss functions, proposed a differentiable surrogate loss function. result Gradient paths of the proposed surrogate Fβ loss function approximate the gradient paths of the Fβ score. Bayesian Optimization (BO) is a framework for black-box optimization that is especially suitable for expensive cost functions. Among the main parts of a BO algorithm, the acquisition function is of fundamental importance, since it guides the optimization algorithm by translating the uncertainty of the regression model …
Label embedding (LE) is an important family of multi-label classification algorithms that digest the label information jointly for better performance. Different real-world applications evaluate performance by different cost functions of interest. Current LE algorithms often aim to optimize one specific cost function, b…
A new Universal Activation Function improves performance across various machine learning tasks.
problem Achieving near optimal performance in different machine learning tasks.
method Optimization algorithms evolve the UAF's parameters to match the optimal activation function for each task.
result The UAF converges to near optimal performance in classification, quantification, and reinforcement learning tasks.
Goldilocks activation functions improve neural network performance.
problem Improving neural network performance and understanding signal transformation.
method Introducing Goldilocks activation functions that locally deform input signals.
result Goldilocks networks outperform or match SELU and RELU on CIFAR-10 and CIFAR-100 datasets.
Bayesian optimization is a sample-efficient method for black-box global optimization. How- ever, the performance of a Bayesian optimization method very much depends on its exploration strategy, i.e. the choice of acquisition function, and it is not clear a priori which choice will result in superior performance. While …
This paper explores optimising acquisition functions in Bayesian optimisation.
problem Optimising acquisition functions in Bayesian optimisation is challenging due to their non-convex nature.
method The authors derive compositional forms for acquisition functions and use them to recast maximisation as a compositional optimisation problem.
result The compositional approach to maximising acquisition functions shows empirical advantages across various tasks.
Develops a new method to learn kernels directly over function space.
problem Learning kernels for flexible function approximators.
method Functional Kernel Learning (FKL) using transformed Gaussian processes over spectral densities.
result Direct inference of functional posteriors over kernels enables rich representations.
LSH methods extend to function spaces for efficient similarity search.
problem Efficient similarity search in function spaces.
method Locality-sensitive hashing (LSH) extended to Lp spaces using function approximation or Monte Carlo techniques. result An LSH family for Wasserstein distance over continuous probability distributions.
Deep learning enhances options hedging performance.
problem Improving delta hedging for options using neural networks.
method Learning residuals between hedging function and implied Black-Scholes delta using neural networks.
result Deep learning significantly improves hedging performance, often by more than 100%.
Stochastic models analyze traffic network performance.
problem Evaluate traffic system performance.
method Stochastic cell transmission models, preference functionals, Gaussian process regression.
result Illustrated in two case studies.
FunBO uses LLMs to discover effective acquisition functions for Bayesian optimization.
problem Designing optimal acquisition functions for Bayesian optimization across diverse problems.
method FunBO leverages FunSearch, an LLM, to learn and evaluate new acquisition functions.
result FunBO discovers acquisition functions that generalize well and outperform existing methods.
It is widely conjectured that the reason that training algorithms for neural networks are successful because all local minima lead to similar performance, for example, see (LeCun et al., 2015, Choromanska et al., 2015, Dauphin et al., 2014). Performance is typically measured in terms of two metrics: training performanc…
We introduce a new class of forward performance processes that are endogenous and predictable with regards to an underlying market information set and, furthermore, are updated at discrete times. We analyze in detail a binomial model whose parameters are random and updated dynamically as the market evolves. We show tha…
The scalability of submodular optimization methods is critical for their usability in practice. In this paper, we study the reducibility of submodular functions, a property that enables us to reduce the solution space of submodular optimization problems without performance loss. We introduce the concept of reducibility…
New ODE-Block handles stateful layers with continuous-in-depth functions using basis functions.
problem Handling stateful layers in ODE-Nets.
method Formulate ODE-Block using continuous-in-depth functions with basis function expansions.
result Enables state-of-the-art performance and reduces memory footprint.
Sharp bounds on binary model inference performance.
problem High-dimensional inference in binary models.
method Convex empirical risk minimization, sharp asymptotics, optimal performance bounds.
result Sharp predictions and optimal performance bounds for binary models.
A survey is performed of various Multi-Armed Bandit (MAB) strategies in order to examine their performance in circumstances exhibiting non-stationary stochastic reward functions in conjunction with delayed feedback. We run several MAB simulations to simulate an online eCommerce platform for grocery pick up, optimizing …
New method improves auto-labeling accuracy by optimizing confidence functions.
problem Overconfident model scores lead to poor TBAL performance.
method Developed a new post-hoc method, Colander, to optimize TBAL confidence functions.
result Achieves up to 60% improvement in coverage over baseline methods.
MUMBO optimizes multiple tasks efficiently, even with low-cost related functions.
problem Efficiently optimizing multiple related functions with low-cost evaluations.
method Derives a novel multi-task version of entropy search.
result Robust performance with low computational overhead across various optimization challenges.
Deep neural networks (DNNs) have become increasingly important due to their excellent empirical performance on a wide range of problems. However, regularization is generally achieved by indirect means, largely due to the complex set of functions defined by a network and the difficulty in measuring function complexity. …
This work explores Target Networks and Functional Regularization in deep Reinforcement Learning.
problem Stability and performance issues in deep Reinforcement Learning due to target value instability.
method Proposes and studies an explicit Functional Regularization approach as a replacement for Target Networks.
result Functional Regularization improves performance and stability compared to Target Networks.
Enhances functional classifier performance with new tree-based methods and unbiased feature importance assessment.
problem Challenges of high-dimensional functional data and biased feature importance assessment.
method Augmented functional classification trees and random forests with ad-hoc conditional permutations for unbiased feature importance.
result Significant enhancement in predictive power of functional classifiers through new feature importance assessment.
Adma proposes a flexible loss function for neural networks.
problem Static loss functions limit neural network performance.
method Introduces a flexible loss function that adapts to ANN complexity and data distribution.
result Flexible loss function achieves state-of-the-art performance.
Due to the intractable partition function, the exact likelihood function for a Markov random field (MRF), in many situations, can only be approximated. Major approximation approaches include pseudolikelihood and Laplace approximation. In this paper, we propose a novel way of approximating the likelihood function throug…
Deep neural networks improve mean function estimation for functional data.
problem Estimating mean functions of functional data.
method Deep neural networks with ReLU activation, sparsely connected.
result Achieves optimal nonparametric convergence rate in empirical norm.
A reinforcement learning framework combining value function and tree search planner for strategic and tactical decisions.
problem Strategic and tactical decision-making in discrete environments.
method Combines value function and tree search planner, using uncertainty modeling and risk measurement.
result Improves performance and learning speed on hard exploration environments.
Convolution Neural Networks (CNN) have recently achieved state-of-the art performance on handwritten Chinese character recognition (HCCR). However, most of CNN models employ the SoftMax activation function and minimize cross entropy loss, which may cause loss of inter-class information. To cope with this problem, we pr…
Enhances trading metrics with financially grounded loss functions.
problem Challenges in financial deep learning, especially interpretability.
method Introduces loss functions derived from finance metrics and turnover regularization.
result Proposed loss functions outperform traditional methods in trading metrics.
Study dynamic risk measures and performance indices using distortion functions.
problem Investigate time consistency of dynamic risk measures and performance indices generated by distortion functions.
method Analyze dynamic coherent risk measures (DCRMs) and dynamic weighted value at risk measures, proving their equivalence. Establish properties of families of DCRMs generated by distortion functions and define corresponding dynamic coherent acceptability indices (DCAIs). Examine time consistency of DCRMs and DCAIs.
result DCRM generated by distortion functions are sub-martingale time consistent but not super-martingale time consistent and not weakly acceptance time consistent.
Improved hypothesis testing and change-point detection using diffusion-based methods.
problem Limited power of score-based hypothesis tests and change-point detection.
method Extending score-based Fisher divergence to diffusion-divergence by multiplying score functions with a matrix-valued function or weight matrix.
result Theoretical quantification and demonstration of optimal performance of diffusion-based algorithms.
Paper adapts CVS method for TL in functional linear regression.
problem Improving estimation and prediction in scalar-on-function regression.
method Adapts control variates method for transfer learning.
result Establishes theoretical connection between O-TL and CVS-based TL.
Study reveals Data Shapley's inconsistent performance in data selection tasks.
problem Inconsistency of Data Shapley's performance in data selection across different settings.
method Hypothesis testing framework and identification of utility functions.
result Data Shapley's performance is no better than random selection without specific constraints.