The paper aims to improve reproducibility of machine learning experiments.
problem Lack of reproducibility in machine learning experiments.
method Investigates factors affecting reproducibility and proposes FAIR data practices.
result Preliminary results show the role of ProvBook in capturing and comparing provenance.
DeepDIVA simplifies reproducible deep learning experiments.
problem Difficulty in reproducing deep learning research results.
method A framework for easy experimentation and reproduction.
result Facilitates sharing and easy experimentation of experiments.
Proposes a method to improve neural architectures reproducibly.
problem Lack of reproducibility in Neural Architecture Transformer (NAT).
method Differentiable Neural Architecture Transformation (DNAT).
result DNAT outperforms NAT and is applicable to various models and datasets.
A modular framework for knowledge distillation simplifies experiments and reproducibility.
problem Difficulty in reproducing knowledge distillation studies due to lack of generalized frameworks.
method A configuration-driven PyTorch framework for knowledge distillation studies.
result Demonstrates efficient training strategies and various knowledge distillation methods.
Replication confirms CGD's effectiveness in competitive games.
problem Reproducibility of a novel Nash equilibrium algorithm.
method Replicated experiments and provided Python implementation.
result CGD avoids oscillatory and divergent behaviours.
ModSSC unifies semi-supervised classification for various data types.
problem Fragmented support for semi-supervised classification across different methods, settings, and data types.
method ModSSC is a modular Python framework that supports reproducible and controlled experimentation for semi-supervised classification on heterogeneous data.
result ModSSC enables systematic comparison of semi-supervised learning across various datasets and model backbones.
This report has several purposes. First, our report is written to investigate the reproducibility of the submitted paper On the regularization of Wasserstein GANs (2018). Second, among the experiments performed in the submitted paper, five aspects were emphasized and reproduced: learning speed, stability, robustness ag…
Paper reproduces and enhances a method for cross-lingual word embeddings.
problem Creating robust cross-lingual mappings of word embeddings without supervision.
method Reproduces and enhances a self-learning method with grid search for hyperparameters.
result Model's robustness is demonstrated across four new languages.
ERICA assesses reproducibility in cluster analysis.
problem Lack of a unified framework for evaluating cluster analysis replicability.
method ERICA (iterative clustering assignments) method to quantify replicability.
result Demonstrates ERICA's ability to identify reproducible cluster structure.
Study confirms improved performance of Self-Critique and Adapt method.
problem Improving performance of MAML++ method.
method Self-Critique and Adapt (SCA) method.
result SCA method improves performance of MAML++.
SLM Lab is a framework for reproducible RL research with modular algorithms.
problem Reproducibility in deep reinforcement learning.
method Modular software framework for RL algorithms, synchronous/asynchronous execution, hyperparameter search, result analysis.
result Comprehensive benchmark and novel RL algorithms (e.g., discrete-AC variant, hybrid training method).
catalyst.RL simplifies RL research, enabling reproducibility and efficiency.
problem Difficulty in reproducing and comparing RL algorithms.
method Open-source framework with distributed training, flexible configurations, and efficient RL algorithms.
result Demonstrated effectiveness on AI for Prosthetics Challenge, achieving 3rd place.
Teaches reproducible research to medical students and postgrads.
problem Lack of reproducibility in medical research practices.
method Designed and delivered a lecture series on reproducible research.
result Encountered practical obstacles in reproducing a published analysis.
BayesBag improves reproducibility of Bayesian inference under model misspecification.
problem Bayesian posteriors can be unreliable and inconsistent under model misspecification.
method Apply bagging to the Bayesian posterior to improve reproducibility.
result Bagged posteriors typically satisfy reproducibility criteria under misspecification.
KEEN Universe provides reproducible and transferable knowledge graph embeddings.
problem Lack of reproducibility and transferability in KGE experiments.
method Developed an ecosystem with Python packages for reproducible and transferable KGEs.
result KEEN Universe facilitates sharing of trained KGE models across different fields.
New optimizer closes gap between adaptive methods and SGD for deep learning.
problem Degraded generalization performance of adaptive gradient methods compared to SGD.
method Introducing a new optimization algorithm with a tunable hyperparameter p.
result The new algorithm bridges the gap between adaptive methods and SGD with momentum.
CK simplifies ML model deployment and reproducibility with open APIs and DevOps.
problem Making ML models reproducible and deployable across different environments.
method Decompose complex systems into reusable sub-components with unified APIs and DevOps principles.
result Automatically co-design and optimize ML models for speed, accuracy, energy, and size.
LLM agents discover cryptocurrency factors under reproducible constraints.
problem Flexibility of LLM agents in empirical discovery leads to uncontrolled search.
method Sequential hypothesis search with fixed data splits and portfolio tests.
result Ridge-combined portfolio achieves 44.55% annualized return in out-of-sample period.
Authors provide a fair comparison of GNNs for graph classification.
problem Lack of reproducibility and rigorousness in experimental procedures for GNNs.
method Controlled and uniform framework with over 47,000 experiments.
result GNNs do not fully exploit structural information on some datasets.
We consider the problem of learning a set from random samples. We show how relevant geometric and topological properties of a set can be studied analytically using concepts from the theory of reproducing kernel Hilbert spaces. A new kind of reproducing kernel, that we call separating kernel, plays a crucial role in our…
A mesh-free method solves continuum-marginal optimal transport problems.
problem Recovering minimum-energy velocity fields from time-continuous probability marginals.
method Embeds weak continuity equation in a reproducing kernel Hilbert space, optimizing with mini-batch stochastic methods.
result Accurately recovers drift and maintains marginal consistency in synthetic experiments.
Study evaluates 21 KG embedding models, highlighting model architecture, training approach, and loss function importance.
problem Difficulty in fair and thorough comparisons of knowledge graph embedding models due to heterogeneity in implementations, training, and evaluation.
method Reimplemented and evaluated 21 models in PyKEEN, providing insights into reproducibility and best practices.
result Best practices and configurations for each model were identified, showing the importance of model architecture, training approach, and loss function.
This work proves L2-regularized ERM controls smCE without post-hoc correction.
problem Calibration of predicted probabilities in machine learning models.
method Canonical L2-regularized empirical risk minimization. result Theoretical proof that smCE is controlled by ERM without post-hoc correction.
A new method learns Hamiltonian functions from noisy data.
problem Learning Hamiltonian functions from noisy observations.
method Structure-preserving kernel ridge regression method.
result The method yields excellent numerical performances.
CogDL simplifies graph deep learning experiments and benchmarks.
problem Challenges in training and evaluating graph neural networks.
method Unified design for training and evaluation, mixed precision training, efficient sparse operators, ease of use.
result CogDL is the most competitive graph library for efficiency and ease of use.
NAS-Bench-101 simplifies neural architecture search by creating a searchable dataset.
problem Difficulty in reproducing NAS experiments due to high computational costs.
method Created a compact, expressive search space with 423k unique architectures, trained and evaluated them on CIFAR-10, and compiled results into a dataset.
result Researchers can evaluate diverse models in milliseconds by querying the pre-computed dataset.
Reproduces IVFS for high-dimensional data structure preservation.
problem Preserving high-dimensional data structure in unsupervised feature selection.
method Inspired by random subset method, IVFS maintains data similarity through topological structure.
result IVFS outperforms SPEC and MCFS on most datasets.
Benchopt automates machine learning benchmarking across languages and hardware.
problem Limited transparency and tedious re-implementation work in machine learning validation.
method A collaborative framework for automating, reproducing, and publishing optimization benchmarks.
result Demonstrates practical findings that highlight the importance of details in machine learning validation.
Study on reproducibility in optimization with bounds on limits.
problem Limits of reproducibility in noisy or error-prone optimization procedures.
method Defined a quantitative measure of reproducibility and analyzed convex optimization settings.
result Revealed a fundamental trade-off between computation and reproducibility.
Embedding RL policies in RKHS for robustness and theoretical guarantees.
problem Stability and theoretical guarantees in RL policy representation.
method Low-dimensional embedding of RL policies in RKHS.
result Embedded policies maintain high return with strong theoretical guarantees.
New method stabilizes machine learning predictions across random seeds.
problem Machine learning predictions vary across random seeds, causing instability.
method Introduces adaptive cross-bagging to eliminate seed dependence.
result Adaptive cross-bagging achieves targeted stability in debiased machine learning.
Survey of determinism issues in financial AI systems.
problem Vulnerabilities in reproducibility of financial AI systems.
method Literature review and first-party experiments on public financial datasets.
result Proposed a layered evaluation framework linking modality-specific metrics to audit readiness.
AL methods show inconsistent performance gains over random sampling, highlighting variability in neural network-based approaches.
problem Inconsistent performance of AL methods using neural networks.
method Different types of AL algorithms (uncertainty based, diversity based, and committee based) were tested under identical experimental settings.
result AL methods do not consistently outperform random sampling, revealing variability in performance metrics.
Generative model uses ODEs and RKHSs for measure matching.
problem Minimum divergence generative modeling and sampling.
method Diffeomorphic matching and image registration principles applied to ODEs and RKHSs.
result Theoretical error bounds and extensive numerical experiments demonstrate the method's properties and applicability.
Deep RL agents vary significantly in Atari environments.
problem Challenges in reproducibility due to stochasticity.
method Experiments with OpenAI Baselines agents.
result Variability in agent performance is significant and underreported.
Kernel based methods have shown effective performance in many remote sensing classification tasks. However their performance significantly depend on its hyper-parameters. The conventional technique to estimate the parameter comes with high computational complexity. Thus, the objective of this letter is to propose an fa…
Paper creates benchmarks for neural hyperparameter search.
problem Difficulty in comparing HPO methods due to high computational costs.
method Developed benchmarks for a feed forward neural network on four regression datasets.
result Exhaustive comparison of HPO methods on the benchmarks.
Survival analysis models research reproducibility, offering new insights.
problem Reproducibility crisis in machine learning research.
method Survival analysis to model reproducibility as a continuous process.
result Survival analysis provides deeper insights into research reproducibility.
Clarifies the scope of 'reproducibility' in AI and ML.
problem Unclear meaning of 'reproducibility' in AI and ML.
method Analyzes the community's works on reproducibility.
result Identifies eight general topic areas of reproducibility.
This chapter introduces reproducibility in machine learning for medical imaging.
problem Lack of reproducibility in machine learning for medical imaging.
method Distinguishes and defines types of reproducibility, outlines requirements, and discusses utility.
result Discussion on benefits and a plea for a non-dogmatic approach to reproducibility.
Laplace kernel and Neural Tangent Kernels are shown to be nearly identical for normalized data.
problem Understanding the similarity between Laplace and Neural Tangent Kernels.
method Theoretical analysis and experiments on normalized data.
result Laplace kernel and Neural Tangent Kernels have nearly identical eigenfunctions and RKHS for normalized data.
New algorithms ensure reproducibility and optimal convergence in convex optimization.
problem Trade-off between reproducibility and convergence rate in convex optimization.
method Regularization-based algorithms for smooth convex minimization and minimax optimization.
result Achieves optimal reproducibility and near-optimal gradient complexity for various oracle settings.
New budget quantifies drift in closed-loop learning, improving reproducibility.
problem Characterizing statistical learning under distributional drift in closed-loop settings.
method Introduces an intrinsic drift budget CT quantifying cumulative information-geometric motion of the data distribution. result Proves a drift-feedback bound of order T−1/2+CT/T for prequential reproducibility, up to controlled second-order remainder terms. Develop a comprehensive theory for regularized M-estimation in reproducing kernel Hilbert spaces.
problem Regularized M-estimation in reproducing kernel Hilbert spaces
method Existence and measurability of the estimator, sharp rates of convergence
result New rates for tensor product Sobolev spaces
In this paper we solve support vector machines in reproducing kernel Banach spaces with reproducing kernels defined on nonsymmetric domains instead of the traditional methods in reproducing kernel Hilbert spaces. Using the orthogonality of semi-inner-products, we can obtain the explicit representations of the dual (nor…
NeurIPS 2019 program improves reproducibility in machine learning.
problem Ensuring machine learning research results are reproducible and reliable.
method Code submission policy, reproducibility challenge, and checklist integration.
result Improved reproducibility standards across the machine learning community.
The paper improves nonparametric confidence bands for band-limited functions.
problem Constructing nonparametric simultaneous confidence bands with nonasymptotic and distribition-free guarantees.
method Based on Paley-Wiener reproducing kernel Hilbert spaces, the paper relaxes assumptions, improves noise estimation, and tightens constraints.
result Enhanced confidence bands with improved efficiency and tighter constraints.
AlphaForgeBench evaluates LLMs as quantitative researchers, not trading agents, to address instability in financial decision-making.
problem Behavioral instability of LLMs in sequential decision-making under financial uncertainty.
method Proposes AlphaForgeBench, a framework that requires LLMs to generate executable alpha factors and compose factor-based trading strategies.
result Eliminates execution-induced instability and provides a rigorous benchmark for evaluating financial reasoning.