YAHPO Gym introduces a new benchmark for evaluating hyperparameter optimization methods.
problem Evaluating and comparing hyperparameter optimization methods on well-curated benchmark suites.
method Surrogate-based benchmark collection of 14 scenarios, each with multi-fidelity and multi-objective hyperparameter optimization problems.
result Surrogate-based benchmarks produce more faithful results than tabular benchmarks.
Proposes a new framework for optimizing utility with state-dependent benchmarks.
problem Various interpretations of benchmarks in utility functions.
method General framework of state-dependent utility optimization with stochastic benchmarks.
result Provides optimal solutions and addresses issues of well-definedness and feasibility.
Framework benchmarks optimizers on multiple criteria.
problem Benchmarking optimizers across diverse test functions.
method Union-free generic depth function for partial orders/rankings.
result Identifies central and outlying rankings of optimizers.
Study optimizes portfolio to minimize relative drawdown duration, penalizing unfavorable performance states.
problem Minimizing relative drawdown duration in portfolio optimization relative to a benchmark.
method Introduces a benchmark-relative drawdown-duration criterion penalizing unfavorable performance states. Uses a one-dimensional Markovian representation and Hamilton-Jacobi-Bellman equation.
result Derives explicit projection-based characterization of the optimal feedback control and identifies geometric settings for unique strong solutions.
Study benchmarks TSC algorithms in distinguishing diffusions using the likelihood ratio test.
problem Benchmarking optimality of TSC algorithms in distinguishing diffusion processes.
method Proposes to benchmark TSC algorithms using the likelihood ratio test (LRT).
result LRT benchmarks are computationally efficient and can be applied to various time series types.
Optimal benchmark design varies based on costs in financial manipulation.
problem Manipulation of price benchmarks in finance.
method Analyzes empirical pattern and cost structures to determine optimal benchmark design.
result The optimal benchmark depends on the relative sizes of fixed and variable costs.
The optimization of algorithm (hyper-)parameters is crucial for achieving peak performance across a wide range of domains, ranging from deep neural networks to solvers for hard combinatorial problems. The resulting algorithm configuration (AC) problem has attracted much attention from the machine learning community. Ho…
Olympus benchmarks optimization algorithms for noisy experiments.
problem Benchmarking optimization algorithms on realistic experimental scenarios is challenging.
method Introduces Olympus, a software package for benchmarking optimization algorithms on synthetic experiments.
result Mitigates barriers in benchmarking optimization algorithms on realistic experimental scenarios.
BAT benchmark for autobidding tasks in RTB auctions.
problem Lack of comprehensive datasets and benchmarks for autobidding.
method Developed a benchmark for two auction formats, implemented robust baselines.
result Provides a framework for developing and refining autobidding algorithms.
Because the choice and tuning of the optimizer affects the speed, and ultimately the performance of deep learning, there is significant past and recent research in this area. Yet, perhaps surprisingly, there is no generally agreed-upon protocol for the quantitative and reproducible evaluation of optimization strategies…
Paper introduces benchmark-neutral pricing for long-term contracts.
problem High prices of long-term contracts under risk-neutral pricing.
method Uses growth optimal portfolio as numeraire and new pricing measure.
result Identifies minimal possible prices for contingent claims.
Due to the high computational demands executing a rigorous comparison between hyperparameter optimization (HPO) methods is often cumbersome. The goal of this paper is to facilitate a better empirical evaluation of HPO methods by providing benchmarks that are cheap to evaluate, but still represent realistic use cases. W…
Optimal asset allocation strategy outperforms stochastic benchmark.
problem Achieving higher terminal wealth than a stochastic benchmark.
method Data-driven Neural Network optimization framework for dynamic asset allocation.
result Optimal adaptive strategy outperforms benchmark with higher median and right-skewed terminal wealth.
This work introduces benchmarks for evaluating nanophotonic structures in design simulations.
problem Design and understanding of nanophotonic structures for various applications.
method Development of frameworks and benchmarks for evaluating nanophotonic structures in parametric design problems.
result Strategic use of evaluation fidelity in enhancing structure designs.
New benchmark protocol evaluates neural network optimizers for efficiency and data shift sensitivity.
problem Benchmarking neural network optimizers with hyperparameter complexity and data shift sensitivity.
method Proposed a new evaluation protocol combining end-to-end and data-addition training efficiency, using bandit hyperparameter tuning and human study validation.
result No clear winner across all tasks, highlighting the complexity of optimizer performance.
As Deep Learning (DL) models have been increasingly used in latency-sensitive applications, there has been a growing interest in improving their response time. An important venue for such improvement is to profile the execution of these models and characterize their performance to identify possible optimization opportu…
Combines absolute and relative wealth in portfolio optimization with power utility functions.
problem Optimizing portfolios with both absolute and relative wealth considerations.
method Integrates power utility functions for absolute and relative wealth, considering multiple benchmarks.
result Obtains an explicit solution for portfolio optimization combining absolute and relative wealth.
Study benchmarks LLMs in portfolio optimization tasks.
problem Evaluate financial decision-making of LLMs.
method Mathematically explicit portfolio optimization problems with multiple-choice questions.
result Distinct performance patterns among LLMs in different financial tasks.
L2O uses machine learning to design optimization methods.
problem Designing efficient optimization methods for specific problem distributions.
method Data-driven approach to automate optimization method design.
result L2O methods are practical for specific problem distributions but fail on out-of-distribution problems.
Survey and benchmark high-dimensional Bayesian optimization of discrete sequences.
problem Heterogeneous experimental set-ups and technical barriers in high-dimensional Bayesian optimization of discrete sequences.
method Unified framework and software libraries to test and benchmark methods.
result Unified framework and software libraries for testing and benchmarking high-dimensional Bayesian optimization methods.
Investors optimize their portfolios within a Wasserstein ball to match a benchmark's risk profile.
problem Optimizing portfolio performance while maintaining risk proximity to a benchmark.
method Optimal dynamic strategy selection based on minimizing distortion risk measures within a Wasserstein ball.
result An optimal dynamic strategy exists and can be calculated through isotonic projections.
Study optimal consumption with relaxed benchmarks and drawdown constraints.
problem Optimal consumption under relaxed benchmark tracking and consumption drawdown constraint.
method Transformed stochastic control problem into regular control problem with state-control constraints, then solved using dual transform and optimal consumption behavior.
result Closed-form solution for optimal investment and consumption in feedback form.
Enhances portfolio optimization under uncertainty using robust multi-objective methods.
problem Uncertainties in real-world portfolio optimization scenarios.
method Robust multi-objective optimization with benchmark comparisons.
result More reliable and adaptable portfolio strategies for market uncertainties.
Several recent papers have examined generalization in reinforcement learning (RL), by proposing new environments or ways to add noise to existing environments, then benchmarking algorithms and model architectures on those environments. We discuss subtle conceptual properties of RL benchmarks that are not required in su…
Flat-minima optimizers improve neural network generalization.
problem Improving neural network generalization performance.
method Stochastic Weight Averaging (SWA) and Sharpness-Aware Minimization (SAM).
result Surprising findings from loss surface analysis and broad benchmarking.
MLPerf, an emerging machine learning benchmark suite strives to cover a broad range of applications of machine learning. We present a study on its characteristics and how the MLPerf benchmarks differ from some of the previous deep learning benchmarks like DAWNBench and DeepBench. We find that application benchmarks suc…
Study finds average 2.02 bps loss in automated market maker routing.
problem Measuring sub-optimality in automated market maker routing.
method Three reproducible optimal benchmarks: SCO, FVO, G-FVO; bisection-based algorithm for optimal routing.
result Average 2.02 bps loss per trade, \$24 million total loss.
Benchopt automates machine learning benchmarking across languages and hardware.
problem Limited transparency and tedious re-implementation work in machine learning validation.
method A collaborative framework for automating, reproducing, and publishing optimization benchmarks.
result Demonstrates practical findings that highlight the importance of details in machine learning validation.
This work introduces a new benchmark to compare neural network training algorithms.
problem Lack of reliable benchmarks to compare training algorithms effectively.
method Developed a new benchmark called AlgoPerf: Training Algorithms benchmark.
result Demonstrated the feasibility of the benchmark and set a provisional state-of-the-art.
Gradient matching method improves domain generalization across various datasets.
problem Machine learning's inability to generalize to unseen domains.
method Inter-domain gradient matching objective and first-order algorithm Fish.
result Fish method produces competitive results and surpasses baselines on 4 datasets.
This study benchmarks fifteen deep learning optimizers and identifies a subset that generally performs well.
problem Choosing the best optimizer in deep learning is challenging and often based on anecdotes.
method An extensive, standardized benchmark of fifteen popular optimizers, analyzing over 50,000 runs.
result A subset of optimizers and parameter choices generally leads to competitive results.
The paper solves a control problem using reflections to track a benchmark process.
problem Optimal consumption with a benchmark process that grows over time.
method Introduced two auxiliary state processes with reflections to transform the problem into a more tractable form.
result Established the existence of a unique classical solution to the dual PDE.
Optimal portfolio tracking with dynamic capital injection into a ratcheting benchmark.
problem Optimizing a portfolio's performance by dynamically adding capital to a non-decreasing benchmark.
method Formulated as an unconstrained control problem with a running maximum cost, transformed into an auxiliary problem with a nonlinear HJB equation, solved using probabilistic representation and stochastic flow analysis.
result Established the existence of a unique classical solution to the HJB equation, providing feedback optimal portfolio strategies.
The paper introduces a new divergence for portfolio management to outperform a benchmark.
problem Maximizing expected utility of outperformance over a benchmark with constraints.
method Uses α-Bregman-Wasserstein divergence to penalize underperformance more than overperformance. result Proves existence and uniqueness of optimal portfolio strategy and conditions for constraints binding.
A new optimizer, MVO, improves nonlinear regression performance.
problem Finding optimal coefficients in nonlinear regression models.
method Multi-Verse Optimizer (MVO) compared to Particle Swarm Optimizer (PSO).
result MVO statistically outperforms PSO in 10 nonlinear regression problems.
Automated HPO design using Bayesian optimization and benchmarking.
problem Designing effective hyperparameter optimization algorithms is manual and lacks systematic understanding.
method Formalized space of HPO candidates, Bayesian optimization for search, ablation analysis.
result Simple configurations can perform well in HPO, especially with right parameters.
This study benchmarks AI agents for personalized retail promotions using simulations.
problem Optimizing coupon targeting for sparse customer purchase events.
method Comprehensive simulations of customer shopping behaviors; training RL agents on batch data.
result Contextual bandit and deep RL methods outperform static policies in sparse reward environments.
Study proposes a neural network approach for high inflation investment portfolios with leverage constraints.
problem Optimizing investment portfolios with high inflation and bounded leverage constraints.
method Formulated an optimal control problem, established a closed-form solution, and developed a novel LFNN approach.
result The LFNN strategy outperforms a passive benchmark by about 200 bps with a high probability of success.
We address the problem of learning to benchmark the best achievable classifier performance. In this problem the objective is to establish statistically consistent estimates of the Bayes misclassification error rate without having to learn a Bayes-optimal classifier. Our learning to benchmark framework improves on previ…
We introduce COCO, an open source platform for Comparing Continuous Optimizers in a black-box setting. COCO aims at automatizing the tedious and repetitive task of benchmarking numerical optimization algorithms to the greatest possible extent. The platform and the underlying methodology allow to benchmark in the same f…
We consider the optimal solutions to the trade execution problem in the two different classes of i) fully adapted or adaptive and ii) deterministic or static strategies, comparing them. We do this in two different benchmark models. The first model is a discrete time framework with an information flow process, dealing w…
Investigate using LETFs to outperform benchmarks, finding them more likely to succeed.
problem The controversy and popularity of LETFs in constructing portfolios.
method Systematic investigation using IR-optimal strategies with LETFs and VETFs, including neural network-based approaches.
result IR-optimal strategies with LETFs outperform benchmarks and achieve partial stochastic dominance.
The paper extends Merton's problem by adding benchmark tracking, finding optimal strategies.
problem Maximizing consumption utility with a trade-off against benchmark performance.
method Developed a convex duality theorem and derived optimal strategies for specific cases.
result Found optimal portfolio and consumption strategies for CRRA utility and geometric Brownian motion benchmarks.
The paper shows that benchmark-neutral pricing minimizes option prices.
problem Pricing extreme-maturity European put options on diversified indices.
method Benchmark-neutral pricing applied to a drifted time-transformed squared Bessel process.
result Benchmark-neutral price is the minimal possible price, risk-neutral price is more expensive.
Machine learning (ML) needs industry-standard performance benchmarks to support design and competitive evaluation of the many emerging software and hardware solutions for ML. But ML training presents three unique benchmarking challenges absent from other domains: optimizations that improve training throughput can incre…
New optimizer MARS-M combines variance reduction with Muon for faster LLM training.
problem Training large-scale neural networks efficiently.
method Integrates MARS variance reduction with Muon optimizer.
result MARS-M converges to a first-order stationary point at a rate of ildeO(T−1/3). Optimistic Mirror Descent framework improves bidding strategies in non-stationary first-price auctions.
problem Optimizing bidding strategies in non-stationary first-price auctions.
method Introducing Optimistic Mirror Descent (OMD) framework with novel optimism configuration.
result Minimax-optimal dynamic regret rates achieved for non-stationary first-price auctions.
Portfolio management problems are often divided into two types: active and passive, where the objective is to outperform and track a preselected benchmark, respectively. Here, we formulate and solve a dynamic asset allocation problem that combines these two objectives in a unified framework. We look to maximize the exp…