Research
On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

168,695 papers · 148 categories

Trend · papers per month

0.3%0.5%0.8%0.7% · May 202619922001200920172026
48 results for testbed

The Neural Testbed evaluates joint predictions of neural agents, revealing their limitations.

problem Evaluating the quality of joint predictions generated by neural agents.
method Developed an open-source benchmark (The Neural Testbed) to assess agents' marginal and joint predictions.
result Popular Bayesian deep learning agents perform poorly on joint predictions, even with accurate marginal predictions.

Study shows current image classification models lack robustness to real-world dataset shifts.

problem Robustness of current image classification models to natural distribution shifts in real datasets.
method Evaluation of 204 ImageNet models in 213 different test conditions.
result Little to no transfer of robustness from synthetic to natural distribution shifts.

SMEs provide a transparent testbed for RL evaluation.

problem Lack of precise, white-box diagnostics in RL environments.
method Synthetic Monitoring Environments (SMEs) with fully configurable task characteristics and known optimal policies.
result SMEs allow for precise evaluation of RL algorithms, revealing the impact of specific environmental properties.

The paper examines domain generalization algorithms and finds empirical risk minimization performs well.

problem Comparing domain generalization algorithms is difficult due to inconsistent experimental conditions.
method Implemented DomainBed, a testbed for domain generalization with seven datasets and model selection criteria.
result Empirical risk minimization shows state-of-the-art performance across all datasets.

DeepRacing uses neural networks to predict trajectories for autonomous racing in video games.

problem Training algorithms for high-speed autonomous racing in realistic environments.
method Developed a virtual testbed using F1 video games, trained neural networks to predict trajectories and control commands.
result Trajectory prediction outperforms end-to-end control methods in autonomous racing simulations.

Recent research shows that the following two models are equivalent: (a) infinitely wide neural networks (NNs) trained under l2 loss by gradient descent with infinitesimally small learning rate (b) kernel regression with respect to so-called Neural Tangent Kernels (NTKs) (Jacot et al., 2018). An efficient algorithm to c…

2019-10-03abs ↗pdf ↗

Reinforcement learning agents are faced with two types of uncertainty. Epistemic uncertainty stems from limited data and is useful for exploration, whereas aleatoric uncertainty arises from stochastic environments and must be accounted for in risk-sensitive applications. We highlight the challenges involved in simultan…

2019-05-23abs ↗pdf ↗

Paper presents a new dataset for testing causal discovery methods in industrial systems.

problem Lack of real-world datasets for evaluating causal discovery methods on time series data.
method Develops a dataset from an industrial system and its known causal graph.
result Provides a benchmark for evaluating causal discovery methods in complex systems.

We present a general framework for classifying partially observed dynamical systems based on the idea of learning in the model space. In contrast to the existing approaches using model point estimates to represent individual data items, we employ posterior distributions over models, thus taking into account in a princi…

2016-07-07abs ↗pdf ↗

Learning goal-oriented dialogues by means of deep reinforcement learning has recently become a popular research topic. However, commonly used policy-based dialogue agents often end up focusing on simple utterances and suboptimal policies. To mitigate this problem, we propose a class of novel temperature-based extension…

2018-07-02abs ↗pdf ↗

Reinforcement learning (RL) has proven its worth in a series of artificial domains, and is beginning to show some successes in real-world scenarios. However, much of the research advances in RL are often hard to leverage in real-world systems due to a series of assumptions that are rarely satisfied in practice. We pres…

2019-04-29abs ↗pdf ↗

Production recommendation systems rely on embedding methods to represent various features. An impeding challenge in practice is that the large embedding matrix incurs substantial memory footprint in serving as the number of features grows over time. We propose a similarity-aware embedding matrix compression method call…

2019-02-26abs ↗pdf ↗

Deep learning has enabled traditional reinforcement learning methods to deal with high-dimensional problems. However, one of the disadvantages of deep reinforcement learning methods is the limited exploration capacity of learning agents. In this paper, we introduce an approach that integrates human strategies to increa…

2018-06-12abs ↗pdf ↗

Small, carefully crafted perturbations called adversarial perturbations can easily fool neural networks. However, these perturbations are largely additive and not naturally found. We turn our attention to the field of Autonomous navigation wherein adverse weather conditions such as fog have a drastic effect on the pred…

2020-01-16abs ↗pdf ↗

This paper introduces a new scalable multi-objective deep reinforcement learning (MODRL) framework based on deep Q-networks. We develop a high-performance MODRL framework that supports both single-policy and multi-policy strategies, as well as both linear and non-linear approaches to action selection. The experimental …

2018-03-08abs ↗pdf ↗

Heretofore, neural networks with external memory are restricted to single memory with lossy representations of memory interactions. A rich representation of relationships between memory pieces urges a high-order and segregated relational memory. In this paper, we propose to separate the storage of individual experience…

2020-02-10abs ↗pdf ↗

Paper uses agent-based simulation to identify investor types in financial markets.

problem Identifying investor types in real financial markets.
method Computational adaptation of PCA with agent-based simulation.
result A reduced set of investor models can approximate financial time series.

Benchmark evaluates financial misinformation detection models, revealing weaknesses without external context.

problem Detecting financial misinformation without external references.
method RFC Bench at paragraph level, two tasks: reference-free detection and comparison-based diagnosis.
result Performance improves with comparative context, revealing model weaknesses in reference-free settings.

PredictionMarketBench benchmarks trading agents on prediction markets.

problem Evaluating trading agents on prediction markets with realistic conditions.
method Deterministic replay of historical data, execution-realistic simulator, agent interface.
result Fee-aware algorithmic strategies outperform naive agents in volatile episodes.

Recent studies have shown that reinforcement learning (RL) is an effective approach for improving the performance of neural machine translation (NMT) system. However, due to its instability, successfully RL training is challenging, especially in real-world systems where deep models and large datasets are leveraged. In …

2018-08-27abs ↗pdf ↗

Knowledge graphs, on top of entities and their relationships, contain other important elements: literals. Literals encode interesting properties (e.g. the height) of entities that are not captured by links between entities alone. Most of the existing work on embedding (or latent feature) based knowledge graph analysis …

2018-02-03abs ↗pdf ↗

This paper analyzes CNNs for malware detection in cloud IaaS.

problem Malware vulnerability in cloud IaaS environments.
method Analysis of Convolutional Neural Networks (CNNs) for online malware detection using process-level performance metrics.
result State-of-the-art DenseNets and ResNets effectively detect malware in online cloud systems.

Clustering consists of grouping together samples giving their similar properties. The problem of modeling simultaneously groups of samples and features is known as Co-Clustering. This paper introduces ROCCO - a Robust Continuous Co-Clustering algorithm. ROCCO is a scalable, hyperparameter-free, easy and ready to use al…

2018-02-14abs ↗pdf ↗

Neural Markov models improve time series analysis by balancing deep learning and classical models.

problem Modeling non-stationary time series with high data sparsity.
method Hybrid approach using neural networks to parameterize stochastic matrices, estimating time-inhomogeneous Markov chains.
result Reduction of Chapman-Kolmogorov discrepancy and superior likelihood in financial markets.

Open problem: Establishing bounds for Cayley-table completion to discover discrete algorithmic axioms.

problem Discovering discrete algorithmic axioms missing in deep learning.
method Cayley-table completion as a testbed for algorithmic complexity minimization.
result Formal exact recovery bounds for Cayley-table completion.

More than 50 years ago Bongard introduced 100 visual concept learning problems as a testbed for intelligent vision systems. These problems are now known as Bongard problems. Although they are well known in the cognitive science and AI communities only moderate progress has been made towards building systems that can so…

2018-04-12abs ↗pdf ↗