FEET protocol evaluates foundation models across three scenarios.
problem Lack of standardized evaluation protocols for foundation models.
method Structured evaluation protocol across three use cases: frozen, few-shot, and fine-tuned embeddings.
result Demonstrates comprehensive assessment of foundation models' effectiveness.
Learning knowledge representation is an increasingly important technology applicable in many domain-specific machine learning problems. We discuss the effectiveness of traditional Link Prediction or Knowledge Graph Completion evaluation protocol when embedding knowledge representation for categorised multi-relational d…
New algorithms for model selection in off-policy evaluation of reinforcement learning.
problem Hyperparameter tuning for off-policy evaluation methods in reinforcement learning.
method Developed new model-free and model-based selectors with theoretical guarantees and a new experimental protocol.
result New model-free selector, LSTD-Tournament, demonstrates promising empirical performance.
New benchmark protocol evaluates neural network optimizers for efficiency and data shift sensitivity.
problem Benchmarking neural network optimizers with hyperparameter complexity and data shift sensitivity.
method Proposed a new evaluation protocol combining end-to-end and data-addition training efficiency, using bandit hyperparameter tuning and human study validation.
result No clear winner across all tasks, highlighting the complexity of optimizer performance.
A new protocol evaluates small machine learning improvements conservatively.
problem Uncertainty in small gains reported in machine learning papers.
method Paired bootstrap protocol with BCa confidence intervals and sign-flip permutation tests.
result Conservative evaluation reduces over-claiming of small improvements.
New federated learning protocols resist Byzantine failures and offer privacy guarantees.
problem Resisting Byzantine failures in federated learning.
method Proposes robust federated learning protocols with optimal statistical rates and privacy guarantees.
result Achieves nearly optimal statistical rates and tight rate in terms of all parameters for strongly convex losses.
This paper critiques flawed MVTS anomaly detection evaluation methods and proposes a simple baseline.
problem Flawed evaluation methods in MVTS anomaly detection research.
method Robust evaluation protocols, including PCA-based baseline.
result Simple PCA-based baseline outperforms many DL approaches.
New protocol evaluates synthetic data for temporal consistency.
problem Synthetic data generators can produce invalid timestamps and trajectories.
method Characterize datasets by four properties, then measure timestamp validity and dynamics.
result Temporal fidelity must be measured, not inferred from static data.
Knowledge bases contribute to many web search and mining tasks, yet they are often incomplete. To add missing facts to a given knowledge base, various embedding models have been proposed in the recent literature. Perhaps surprisingly, relatively simple models with limited expressiveness often performed remarkably well …
Compound Finance optimizes risk metrics for V3 protocol using Chainrisk simulations.
problem Optimizing systemic risks in Compound V3 protocol.
method Millions of Chainrisk simulations to evaluate VaR and LaR, providing parameter adjustments.
result Optimization framework enhances protocol stability.
Protocol for constructing tailored evaluation datasets for semantic models.
problem Evaluation of domain-specific semantic models, focusing on top ranks.
method Adaptive pairwise comparisons, relatedness-based evaluation dataset, metrics, stochastic transitivity model.
result Effectiveness of the proposed dataset construction protocol confirmed.
Benchmark detects decision-time leakage in financial backtests.
problem Detecting decision-time leakage in financial machine-learning backtests.
method Toggles one evaluation convention at a time around a clean t+1-open reference, holding other factors fixed. result Inflation is highly selective, affecting specific features and execution methods.
This study aims to improve communication between fragmented blockchain systems in finance.
problem Inefficient and insecure communication in fragmented blockchain systems.
method Analysis of cross-chain interoperability protocols and their properties.
result Comparison and evaluation of cross-chain interoperability protocols.
Swarm systems constitute a challenging problem for reinforcement learning (RL) as the algorithm needs to learn decentralized control policies that can cope with limited local sensing and communication abilities of the agents. While it is often difficult to directly define the behavior of the agents, simple communicatio…
GraphBench creates a unified benchmark for graph learning tasks.
problem Fragmented benchmarking practices and inconsistent evaluation protocols in graph learning.
method Developed a comprehensive benchmark suite with standardized evaluation protocols.
result Established principled baselines for future research in graph learning.
Since the introduction and the public availability of the \textsc{ucr} time series benchmark data sets, numerous Time Series Classification (TSC) methods has been designed, evaluated and compared to each others. We suggest a critical view of TSC performance evaluation protocols put in place in recent TSC literature. Th…
In this paper, we study the problem of computing U-statistics of degree 2, i.e., quantities that come in the form of averages over pairs of data points, in the local model of differential privacy (LDP). The class of U-statistics covers many statistical estimates of interest, including Gini mean difference, Kendal…
We propose an efficient protocol for decentralized training of deep neural networks from distributed data sources. The proposed protocol allows to handle different phases of model training equally well and to quickly adapt to concept drifts. This leads to a reduction of communication by an order of magnitude compared t…
SIREN protocol corrects optimistic winner's scores in LLM evaluation.
problem Optimistic winner's scores in LLM evaluation due to adaptive benchmarking.
method SIREN protocol that freezes post-search shortlist, separates selection and evaluation, and uses bootstrap for uncertainty quantification.
result SIREN provides valid confidence intervals for procedure-performance curves and deployment conclusions.
Study finds no significant short-term impact on liquidity supply after protocol fees were reduced.
problem Liquidity provider welfare is affected by protocol fees, but the impact on liquidity supply is unclear.
method Used a matched-overlap event-study difference-in-differences design to estimate the liquidity-supply response to take-rate cuts.
result No significant short-term impact on active liquidity or local depth; no change in LP participation or composition.
Graph learning categorizes DeFi services into similar functionalities.
problem Identifying similar financial services in decentralized finance protocols.
method Graph representation learning (GRL) to categorize smart contract blocks into clusters.
result Purity of clustering reaches .888 in the best-case scenario.
Deep reinforcement learning algorithms have shown an impressive ability to learn complex control policies in high-dimensional tasks. However, despite the ever-increasing performance on popular benchmarks, policies learned by deep reinforcement learning algorithms can struggle to generalize when evaluated in remarkably …
The paper examines how decentralized credit curators have taken over risk management from traditional protocols.
problem Risk management in decentralized credit has shifted from centralized protocols to decentralized curators.
method Analysis of ERC 4626 vaults and third-party curators, focusing on capital utilization, concentration, and fee margins.
result Curators have a significant impact on the risk profile of decentralized credit systems, with a small set of curators handling a disproportionate share of system TVL.
Auto.gov uses RL to automate DeFi governance, improving security and profitability.
problem Manual DeFi governance is prone to human bias and financial risks.
method Auto.gov employs a deep Q-network reinforcement learning strategy for semi-automated parameter adjustments. result Auto.gov outperforms traditional governance methods by at least 14% in terms of protocol profitability. Paper explores anchoring for vision models, improving generalization and safety.
problem Anchoring can lead to undesirable shortcuts, limiting generalization.
method Introduced a new anchored training protocol with a regularizer to mitigate undesirable shortcuts.
result Significant performance gains in generalization and safety metrics.
RL Unplugged benchmarks offline RL methods across diverse domains.
problem Evaluate offline reinforcement learning methods without online data collection.
method Proposes a benchmark suite with diverse datasets and detailed evaluation protocols.
result Demonstrates the effectiveness of offline RL methods across various domains.
Study evaluates bias mitigation methods in deep learning, finds they often exploit hidden biases.
problem Deep learning systems learn biases, affecting performance on minority groups.
method Improved evaluation protocol, new dataset, robustness across different tuning distributions.
result Bias mitigation methods often exploit hidden biases, are not robust to multiple forms of bias, and are sensitive to tuning set choice.
Convolutional neural networks (CNNs) have shown promising results on several segmentation tasks in magnetic resonance (MR) images. However, the accuracy of CNNs may degrade severely when segmenting images acquired with different scanners and/or protocols as compared to the training data, thus limiting their practical u…
The paper shows how ignoring temporal context in recommender systems evaluation leads to false confidence, proposing a method to embed temporal context.
problem The discrepancy between offline and online recommender system performance evaluation.
method Proposes a training procedure to embed temporal context into recommender systems and validates its advantage using multi-objective optimization.
result Including temporal context in recommender systems evaluation can improve recall@20 by up to 20%.
A framework for evaluating and benchmarking concept drift detection methods
problem Data stream mining challenged by concept drift
method A novel benchmarking framework
result Fair comparisons of drift detection methods
New study on time series anomaly detection shows overlapping inference improves performance.
problem Heterogeneous evaluation practices and inference procedures in time series anomaly detection.
method Unified training, tuning, and evaluation protocol on TSB-AD benchmark, analyzing overlapping vs. disjoint inference.
result Overlapping inference yields consistent improvements, with average relative gain up to +28%.
A new protocol corrects confounding effects to measure alignment-induced activation shifts accurately.
problem Confounding effects in measuring alignment-induced activation shifts using naive methods.
method Introduces a four-variant decomposition to separate alignment shift from template effects.
result Correctly measures alignment-induced activation shifts, recovering behaviorally active subspace.
FinReflectKG - EvalBench benchmarks financial KG extraction from SEC 10-K filings.
problem Lack of universal benchmark and evaluation framework for financial KG construction.
method Agentic and holistic evaluation principles, deterministic commit-then-justify judging protocol, binary and ordinal evaluations.
result Reflection-based extraction outperforms single-pass extraction in comprehensiveness, precision, and relevance.
This paper improves domain adaptation methods using graph embedding.
problem Alleviating distribution gaps between different data domains.
method Formulate domain adaptation as graph embedding, analyze loss functions, propose rectified evaluation protocol.
result Improved benchmarks on standard datasets demonstrate the effectiveness of the proposed methods.
Maker Protocol manages Dai stable coin on Ethereum blockchain.
problem Managing decentralized finance applications on blockchain.
method Analyzes Maker Protocol's components and governance.
result Maker Protocol is a significant decentralized finance application.
This paper examines interest rates and market efficiency in DeFi loanable funds protocols.
problem Equilibrium of supply and demand for loanable funds in DeFi protocols.
method Review of interest rate mechanisms in Compound, Aave, and dYdX; empirical analysis of market efficiency and inter-connectedness.
result Interest rate rules in DeFi protocols do not always equilibrate supply and demand.
We study the power of interactivity in local differential privacy. First, we focus on the difference between fully interactive and sequentially interactive protocols. Sequentially interactive protocols may query users adaptively in sequence, but they cannot return to previously queried users. The vast majority of exist…
The paper analyzes how mutable blockchain protocols affect miner behavior and strategic stability.
problem The mutability of blockchain protocols undermines long-term planning and cooperative equilibria.
method Integrates Austrian capital theory with repeated game theory to examine miner behavior under different institutional conditions.
result Effective time preference increases when protocol rules are mutable, leading to political rent-seeking and undermining strategic coherence.
Protocol minimizes disclosure in classification tasks.
problem Ensuring minimal disclosure in classification protocols.
method Developed a protocol for multi-party classification that minimizes non-responsive document disclosure.
result Guarantees minimal disclosure of non-responsive documents.
This study measures liquidity risks in Aave, a blockchain lending protocol.
problem Liquidity risks in lending protocols, especially in Aave.
method Measurements of liquidity risks using Aave as a case study, focusing on available liquidity and market concentration.
result Liquidity risks in Aave are volatile and affect the protocol negatively, especially for repeat borrowers.
Hybrid LLM generates synthetic data preserving causal parameters.
problem Synthetic data fails to accurately estimate causal effects.
method Combines model-based covariate synthesis with separately learned propensity and outcome models.
result Hybrid framework ensures causal structure in synthetic data.
Paper presents a risk management framework for blockchain protocols.
problem Blockchain protocol risks affecting DLT and digital assets.
method Developed a comprehensive risk management framework using traditional taxonomy.
result Structured approach to identify, measure, monitor and report blockchain protocol risks.
Study analyzes risk management in Aave and Compound lending protocols, finding v3 better than v2.
problem Risk management in decentralized lending protocols.
method Cross-version and cross-chain analysis using fixed effects model.
result v3 protocols have better risk management, with stronger impact on L2 blockchains.
Feedback alignment methods need to be evaluated for accuracy and gradient cosine similarity.
problem Evaluating feedback alignment methods
method Proposed diagnostic evaluation protocol
result Identified silent failures in standard reporting pair
Distributing Neural Network training is of particular interest for several reasons including scaling using computing clusters, training at data sources such as IOT devices and edge servers, utilizing underutilized resources across heterogeneous environments, and so on. Most contemporary approaches primarily address sca…
Objective: Anemia is a frequent comorbidity in hemodialysis patients that can be successfully treated by administering erythropoiesis-stimulating agents (ESAs). ESAs dosing is currently based on clinical protocols that often do not account for the high inter- and intra-individual variability in the patient's response. …
The paper gives bounds for how long it takes for gossip protocols to spread information in networks.
problem Understanding the diffusion time in asynchronous gossip protocols.
method Provides non-asymptotic bounds for the number of messages needed for consensus in asynchronous gossip protocols.
result Explicit formula and approximation for the number of messages needed for consensus in different types of graphs.
Paper explores how to design federated learning protocols that benefit all participants while maintaining privacy.
problem Privacy concerns undermine the accuracy benefits of federated learning in privacy-sensitive domains.
method The paper provides conditions for mutually beneficial federated learning protocols and designs protocols that maximize total utility and accuracy.
result The paper demonstrates that federated learning can be designed to be mutually beneficial, striking a balance between privacy and model accuracy.