A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
VB-Score evaluates AI systems without ground truth, revealing robustness.
Recommendation systems have been integrated into the majority of large online systems. They tailor those systems to individual users by filtering and ranking information according to user profiles. This adaptation process influences the way users interact with the system and, as a consequence, increases the difficulty …
This paper evaluates deferring systems using causal inference.
Recommendation systems have been integrated into the majority of large online systems to filter and rank information according to user profiles. It thus influences the way users interact with the system and, as a consequence, bias the evaluation of the performance of a recommendation algorithm computed using historical…
The paper shows how ignoring temporal context in recommender systems evaluation leads to false confidence, proposing a method to embed temporal context.
This paper introduces a model of environmental acoustic scenes which adopts a morphological approach by ab-stracting temporal structures of acoustic scenes. To demonstrate its potential, this model is employed to evaluate the performance of a large set of acoustic events detection systems. This model allows us to expli…
Recommendation systems have been integrated into the majority of large online systems to filter and rank information according to user profiles. It thus influences the way users interact with the system and, as a consequence, bias the evaluation of the performance of a recommendation algorithm computed using historical…
Process Monitoring involves tracking a system's behaviors, evaluating the current state of the system, and discovering interesting events that require immediate actions. In this paper, we consider monitoring temporal system state sequences to help detect the changes of dynamic systems, check the divergence of the syste…
Grading function evaluates Bach-style chorales, outperforming human experts.
MARS-Gym framework for marketplaces to train and evaluate recommender systems.
New fairness notion helps identify fair auditors for evaluating decision-support systems.
Recent developments in Neural Relation Extraction (NRE) have made significant strides towards Automated Knowledge Base Construction (AKBC). While much attention has been dedicated towards improvements in accuracy, there have been no attempts in the literature to our knowledge to evaluate social biases in NRE systems. W…
In this paper, we attempt to improve Statistical Machine Translation (SMT) systems on a very diverse set of language pairs (in both directions): Czech - English, Vietnamese - English, French - English and German - English. To accomplish this, we performed translation model training, created adaptations of training sett…
Bayesian approach quantifies uncertainty in LLM evaluations.
Deep-PrAE improves rare-event simulation for black-box systems.
New method for evaluating LLMs reduces bias in open-ended evaluations.
HollowFlow speeds up likelihood evaluation for large-scale models.
In academic literature, recommender systems are often evaluated on the task of next-item prediction. The procedure aims to give an answer to the question: "Given the natural sequence of user-item interactions up to time t, can we predict which item the user will interact with at time t+1?". Evaluation results obtained …
Multiverse analysis helps prevent fairness hacking and evaluate model design decisions.
In this paper, a new deep reinforcement learning based augmented general sequence tagging system is proposed. The new system contains two parts: a deep neural network (DNN) based sequence tagging model and a deep reinforcement learning (DRL) based augmented tagger. The augmented tagger helps improve system performance …
Orpheus simplifies deep learning deployment on edge devices.
There is a growing trend of applying machine learning methods to medical datasets in order to predict patients' future status. Although some of these methods achieve high performance, challenges still exist in comparing and evaluating different models through their interpretable information. Such analytics can help cli…
Machine Learning (ML) is increasingly applied in real-life scenarios, raising concerns about bias in automatic decision making. We focus on bias as a notion of opinion exclusion, that stems from the direct application of traditional ML pipelines to infer subjective properties. We argue that such ML systems should be ev…
The paper shows how shared random seeds can reduce variance in machine learning evaluations.
Many successful applications of computer vision to image or video manipulation are interactive by nature. However, parameters of such systems are often trained neglecting the user. Traditionally, interactive systems have been treated in the same manner as their fully automatic counterparts. Their performance is evaluat…
Regshock visualizes financial risks to help regulators manage systemic shocks.
This paper evaluates financial competitiveness of Indian real estate companies using entropy method.
Improving predictive understanding of Earth system variability and change requires data-model integration. Efficient data-model integration for complex models requires surrogate modeling to reduce model evaluation time. However, building a surrogate of a large-scale Earth system model (ESM) with many output variables i…
While autonomous vehicle (AV) technology has shown substantial progress, we still lack tools for rigorous and scalable testing. Real-world testing, the evaluation method, is dangerous to the public. Moreover, due to the rare nature of failures, billions of miles of driving are needed to statisticall…
As part of the 2016 public evaluation challenge on Detection and Classification of Acoustic Scenes and Events (DCASE 2016), the second task focused on evaluating sound event detection systems using synthetic mixtures of office sounds. This task, which follows the `Event Detection - Office Synthetic' task of DCASE 2013,…
New method evaluates AI stock prediction systems based on decision-making processes.
Benchmark evaluates AI-generated financial QA hallucinations, highlighting system vulnerabilities.
Interpretable Machine Learning (IML) has become increasingly important in many real-world applications, such as autonomous cars and medical diagnosis, where explanations are significantly preferred to help people better understand how machine learning systems work and further enhance their trust towards systems. Howeve…
This research explores the effects of various training settings from Polish to English Statistical Machine Translation system for spoken language. Various elements of the TED parallel text corpora for the IWSLT 2013 evaluation campaign were used as the basis for training of language models, and for development, tuning …
Proposes a method to estimate policy values in reinforcement learning with unmeasured confounders.
Develops a method to simulate rare dangerous events in autonomous systems.
Paper introduces OARF benchmark suite for federated learning systems.
As machine learning systems move from computer-science laboratories into the open world, their accountability becomes a high priority problem. Accountability requires deep understanding of system behavior and its failures. Current evaluation methods such as single-score error metrics and confusion matrices provide aggr…
Evaluation of systemic risk in networks of financial institutions in general requires information of inter-institution financial exposures. In the framework of Debt Rank algorithm, we introduce an approximate method of systemic risk evaluation which requires only node properties, such as total assets and liabilities, a…
Deep learning has undoubtedly offered tremendous improvements in the performance of state-of-the-art speech emotion recognition (SER) systems. However, recent research on adversarial examples poses enormous challenges on the robustness of SER systems by showing the susceptibility of deep neural networks to adversarial …
Meta-reinforcement learning improves fault-adaptive control efficiency.
New metrics CWSA and CWSA+ improve model evaluation under confidence thresholds.
While recent developments in autonomous vehicle (AV) technology highlight substantial progress, we lack tools for rigorous and scalable testing. Real-world testing, the evaluation environment, places the public in danger, and, due to the rare nature of accidents, will require billions of miles in or…
SOCRATES uses LLMs to automate simulation optimization of complex systems.
This paper describes the Speech Technology Center (STC) antispoofing systems submitted to the ASVspoof 2019 challenge. The ASVspoof2019 is the extended version of the previous challenges and includes 2 evaluation conditions: logical access use-case scenario with speech synthesis and voice conversion attack types and ph…
Paper tackles counterfactual sentence detection and evaluation.
The study evaluates how well local explanations align with model predictions.
A new method uses counterfactual learning to improve recommendation system evaluation.