A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
Geospatial ML models need special evaluation methods due to their unique challenges.
This paper introduces a very challenging dataset of historic German documents and evaluates Fully Convolutional Neural Network (FCNN) based methods to locate handwritten annotations of any kind in these documents. The handwritten annotations can appear in form of underlines and text by using various writing instruments…
NIST's CTS Challenge evaluates speaker recognition using telephony data.
CVPR 2020 challenge evaluates continual learning in computer vision.
Study categorizes time series anomaly detection metrics based on evaluation challenges.
The paper evaluates samplers on multi-modal targets, focusing on mode separation and recovery.
This paper describes the Speech Technology Center (STC) antispoofing systems submitted to the ASVspoof 2019 challenge. The ASVspoof2019 is the extended version of the previous challenges and includes 2 evaluation conditions: logical access use-case scenario with speech synthesis and voice conversion attack types and ph…
This paper describes audEERING's submissions as well as additional evaluations for the One-Minute-Gradual (OMG) emotion recognition challenge. We provide the results for audio and video processing on subject (in)dependent evaluations. On the provided Development set, we achieved 0.343 Concordance Correlation Coefficien…
ASEs use surrogate estimation to efficiently evaluate model performance with minimal labels.
Experiments used in current continual learning research do not faithfully assess fundamental challenges of learning continually. Instead of assessing performance on challenging and representative experiment designs, recent research has focused on increased dataset difficulty, while still using flawed experiment set-ups…
This paper proposes a set of criteria to evaluate the objectiveness of explanation methods of neural networks, which is crucial for the development of explainable AI, but it also presents significant challenges. The core challenge is that people usually cannot obtain ground-truth explanations of the neural network. To …
Paper discusses ASD challenge for machine condition monitoring.
Benchmarking off-policy evaluation methods for complex policies.
Evaluating the style of handwriting generation is a challenging problem, since it is not well defined. It is a key component in order to develop in developing systems with more personalized experiences with humans. In this paper, we propose baseline benchmarks, in order to set anchors to estimate the relative quality o…
The optimization of expensive-to-evaluate black-box functions over combinatorial structures is an ubiquitous task in machine learning, engineering and the natural sciences. The combinatorial explosion of the search space and costly evaluations pose challenges for current techniques in discrete optimization and machine …
This paper tackles deep clustering evaluation challenges in high-dimensional data.
Orpheus simplifies deep learning deployment on edge devices.
This paper presents an evaluation framework that attempts to quantify the "degree of realism" of simulated financial time series, whatever the simulation method could be, with the aim of discover unknown characteristics that are not being properly reproduced by such methods in order to improve them. For that purpose, t…
Adaptive framework generates challenging adversarial scenarios for autonomous vehicles.
EWFM trains continuous flows with only energy evaluations, improving sample quality with fewer computations.
Fairness in LLMs is impossible due to inherent technical challenges.
The VoxCeleb Speaker Recognition Challenge 2019 aimed to assess how well current speaker recognition technology is able to identify speakers in unconstrained or `in the wild' data. It consisted of: (i) a publicly available speaker recognition dataset from YouTube videos together with ground truth annotation and standar…
This paper evaluates metrics for graph generative models, addressing common pitfalls.
We present the Voice Conversion Challenge 2018, designed as a follow up to the 2016 edition with the aim of providing a common framework for evaluating and comparing different state-of-the-art voice conversion (VC) systems. The objective of the challenge was to perform speaker conversion (i.e. transform the vocal ident…
Our paper improves uplift model evaluation on randomized controlled trials (RCT) data.
In many data mining applications collection of sufficiently large datasets is the most time consuming and expensive. On the other hand, industrial methods of data collection create huge databases, and make difficult direct applications of the advanced machine learning algorithms. To address the above problems, we consi…
This paper tackles label-efficient evaluation in extreme class imbalance.
We organized a competition on Autonomous Lifelong Machine Learning with Drift that was part of the competition program of NeurIPS 2018. This data driven competition asked participants to develop computer programs capable of solving supervised learning problems where the i.i.d. assumption did not hold. Large data sets w…
BetaExplainer improves GNN interpretability by masking unimportant edges.
VB-Score evaluates AI systems without ground truth, revealing robustness.
Forecasting the movements of stock prices is one the most challenging problems in financial markets analysis. In this paper, we use Machine Learning (ML) algorithms for the prediction of future price movements using limit order book data. Two different sets of features are combined and evaluated: handcrafted features b…
Machine learning (ML) needs industry-standard performance benchmarks to support design and competitive evaluation of the many emerging software and hardware solutions for ML. But ML training presents three unique benchmarking challenges absent from other domains: optimizations that improve training throughput can incre…
This work evaluates deep generative models using RD curves, providing a more comprehensive quality assessment.
This study evaluates Algorand and Ethereum 2.0 for blockchain trilemma challenges.
This paper challenges the current metrics used for evaluating long-term forecasting models.
Reinforcement learning (RL) has proven its worth in a series of artificial domains, and is beginning to show some successes in real-world scenarios. However, much of the research advances in RL are often hard to leverage in real-world systems due to a series of assumptions that are rarely satisfied in practice. We pres…
Paper extends knowledge gradient for preferential BO, overcoming computational challenges.
Bayesian Optimization tackles hidden constraints in architecture optimization.
Semantic code search is the task of retrieving relevant code given a natural language query. While related to other information retrieval tasks, it requires bridging the gap between the language used in code (often abbreviated and highly technical) and natural language more suitable to describe vague concepts and ideas…
Catastrophic forgetting and capacity saturation are the central challenges of any parametric lifelong learning system. In this work, we study these challenges in the context of sequential supervised learning with an emphasis on recurrent neural networks. To evaluate the models in the lifelong learning setting, we propo…
Entity resolution (ER) presents unique challenges for evaluation methodology. While crowdsourcing platforms acquire ground truth, sound approaches to sampling must drive labelling efforts. In ER, extreme class imbalance between matching and non-matching records can lead to enormous labelling requirements when seeking s…
New metrics needed for streaming ML due to delayed labels.
A new framework uses multi-agent reinforcement learning for evaluating policies in two-sided markets.
Evaluation of deep reinforcement learning (RL) is inherently challenging. In particular, learned policies are largely opaque, and hypotheses about the behavior of deep RL agents are difficult to test in black-box environments. Considerable effort has gone into addressing opacity, but almost no effort has been devoted t…
New interface explains contextual bandits to non-experts.
This paper describes our UNet based deep convolutional neural network approach on the Traffic4cast challenge 2019. Challenges task is to predict future traffic flow volume, heading and speed on high resolution whole city map. We used UNet based deep convolutional neural network to train predictive model for the short t…
Deep Reinforcement Learning has shown great success in a variety of control tasks. However, it is unclear how close we are to the vision of putting Deep RL into practice to solve real world problems. In particular, common practice in the field is to train policies on largely deterministic simulators and to evaluate alg…
Spatially-aware metrics improve uncertainty evaluation in segmentation.