Report on challenges and approaches for Multi-Agent RL.
problem Challenges in Multi-Agent RL for cooperative and competitive environments.
method Decentralized Actor, Centralized Critic approach based on Decentralized Partially Observable MDPs.
result Advances in Multi-Agent RL for mixed cooperative and competitive environments.
A decentralized approach for agents to learn and optimize collectively.
problem Challenges in coordinating non-cooperative agents to solve complex sequential decision problems.
method Designing a learning environment where agents learn by trading and optimizing local objectives, leading to a Nash equilibrium.
result Decentralized reinforcement learning algorithms that can handle various decision-making scenarios.
New framework learns swarm behaviors from demonstrations.
problem Applying IRL to large-scale, homogeneous multi-agent systems.
method Introducing swarMDP, reducing multi-agent IRL to single-agent, proposing heterogeneous learning scheme.
result Value functions of swarm agents coincide, enabling single-agent solution.
DORIS algorithm achieves no-regret learning in Markov games with adversarial opponents.
problem Decentralized policy learning in Markov games with nonstationary opponents.
method DORIS algorithm using optimistic hyperpolicy mirror descent.
result Achieves K \sqrt{K} K -regret in general function approximation. This study compares decentralized banks and finds some lack decentralization.
problem Decentralized banks do not fully decentralize transactions as expected.
method Network analysis of transaction data from four banks using core-periphery features.
result MakerDao and Compound are more decentralized than Aave and Liquity.
Systematizes blockchain decentralization taxonomy and metrics.
problem Lack of a unified definition for blockchain decentralization.
method Formulated a taxonomy of five facets and developed metrics.
result Provided comprehensive insights into blockchain decentralization.
Sparse MDP with entropy regularization improves reinforcement learning performance.
problem Improving reinforcement learning policies with sparse and multi-modal distributions.
method Proposes a sparse Markov decision process with causal sparse Tsallis entropy regularization.
result The proposed method achieves a constant performance error bound, outperforming soft MDPs.
Adaptive reduction scheme approximates optimal policy in regularized MDPs.
problem Finding near optimal policy in regularized MDPs with biased solutions.
method Adaptive reduction of regularization parameter λ to approximate optimal policy.
result Iteration complexity reduced for obtaining ε-optimal policy.
Study finds Aave token network has core-periphery structure, with high decentralization predicting better returns.
problem Understanding the actual decentralization in DeFi token transactions on the Ethereum blockchain.
method Applied social network analysis to measure decentralization in Aave token transactions.
result A more decentralized Aave token network predicts higher returns and lower volatility.
We study Exo-MDPs to reduce sample complexity in reinforcement learning.
problem Reducing sample complexity in reinforcement learning for structured MDPs.
method Introducing Exo-MDPs and proving structural equivalence to linear mixture MDPs, establishing regret bounds.
result Proved O ( H 3 / 2 d K ) O(H^{3/2}d\sqrt{K}) O ( H 3/2 d K ) regret bound for Exo-MDPs, matching lower bounds. MakerDAO's governance is centralized despite its decentralized claim.
problem Decentralization illusion in Decentralized Finance (DeFi) governance.
method Empirical analysis using financial, transaction, network, and sentiment indicators.
result Centralized governance impacts Maker protocol and voting power distribution.
Paper establishes new lower bounds for MDPs with changing transition kernels.
problem Minimizing sample complexity and regret in non-stationary MDPs.
method Developed novel lower bounds and constructed hard MDPs.
result Proved Ω ( ( H 3 S A / ε 2 ) log ( 1 / δ ) ) Ω((H^3SA/ε^2)\log(1/δ)) Ω (( H 3 S A / ε 2 ) log ( 1/ δ )) sample complexity lower bound. Decentralized finance uses blockchain for $70B in assets, differing from traditional finance.
problem Ensuring compliance and security in decentralized finance.
method Systematic analysis of legal, economic, security, and privacy aspects.
result Decentralized finance offers unique economic effects and security features.
Study on decentralization in DAOs and its effect on financial efficiency in DeFi.
problem Understanding the impact of decentralization on financial efficiency in blockchain-based governance.
method Analysis using Gini coefficient as an inequality indicator, comparing ROI of token owners.
result Real decentralization affects financial efficiency positively in DeFi.
This paper analyzes voter coalitions in MakerDAO's decentralized governance.
problem Understanding the governance structure and influence of voter coalitions in DAOs.
method Applied clustering algorithm to voting history of MakerDAO to identify voter coalitions.
result The emergence of a dominant voter coalition signals governance centralization in DAOs.
ARL algorithm reduces adversarial MDP to bandit problems for reliable policy learning.
problem Learning reliable policies in non-stationary, adversarial MDPs.
method Adversarial Reinforcement Learning (ARL) algorithm that converts MDP to a sequence of adversarial bandit problems.
result Achieves optimal regret bound of O ( S A T H 3 ) O(\sqrt{SATH^3}) O ( S A T H 3 ) . The paper analyzes stability and generalization of decentralized SGD.
problem Stability and generalization of decentralized stochastic gradient descent.
method Novel formulation of decentralized stochastic gradient descent combined with non/convex optimization theory.
result First stability and generalization guarantees for decentralized stochastic gradient descent.
NFTs raise concerns like scams, racism, and sexism; centralization vs decentralization debate.
problem Concerns and value judgments of stakeholders in NFT market.
method Mixed quantitative and qualitative methods: social media analysis and interviews.
result Identified financial scams, counterfeit NFTs, hacking, and unethical NFTs as major issues.
Deep RL solves combinatorial selection problems with large item spaces.
problem Solving MDPs with large state and action spaces, especially for combinatorial selection.
method Convert S-MDP to IS-MDP, use weight-shared Q-networks to manage state space explosion.
result Our approach effectively handles large item spaces and scales to diverse environments.
Proposes a new algorithm to reduce communication costs in decentralized training.
problem How to apply error-compensated compression to decentralized training.
method Error-compensated stochastic gradient descent for decentralized training.
result Proposed algorithm outperforms existing methods in communication cost reduction.
A decentralized approach for multi-source domain adaptation.
problem Transfer knowledge from multiple related domains to an unlabeled target domain.
method Federated Dataset Dictionary Learning (FedDaDiL) framework, eliminating central server, using Wasserstein barycenters.
result Our decentralized approach effectively adapts source domains to an unlabeled target domain.
New method approximates POMDPs with PB-MDPs, providing error bounds and practical algorithms.
problem Difficulty in solving POMDPs with continuous or hybrid state and observation spaces.
method Bounding particle filtering error and adapting MDP algorithms to POMDPs.
result General theory and practical algorithms for POMDPs with no direct dependence on state and observation space sizes.
D-SPIDER-SFO solves nonconvex optimization problems faster on decentralized networks.
problem Finding a decentralized algorithm with similar convergence rate to SPIDER-SFO.
method Proposed D-SPIDER-SFO, a decentralized variant of SPIDER-SFO.
result Achieves a similar gradient computation cost to centralized SPIDER-SFO.
Decentralized Gaussian processes for multi-agent learning.
problem Training and prediction in multi-agent systems.
method Decentralized ADMM for GP hyper-parameter training and iterative consensus for prediction.
result Subset of agents can perform predictions using covariance-based nearest neighbor selection.
Paper tightens lower bounds on decentralized training complexity.
problem Understanding and optimizing iteration complexity in decentralized training.
method Proved a tight lower bound on iteration complexity and proposed DeTAG algorithm.
result DeTAG achieves the theoretical lower bound with only a logarithmic gap.
This study measures decentralization in blockchain finance governance.
problem Defining and measuring decentralization in blockchain finance applications.
method Aggregating and analyzing empirical data of four finance applications to calculate coefficients for governance token distribution.
result Gauges for objective evaluation of token governance capabilities and limitations.
Paper proposes decentralized annuities for better retirement security.
problem Current pension systems' limitations and fairness issues.
method Theoretical models and fairness concepts analysis.
result Decentralized annuities offer enhanced flexibility and social welfare.
We consider large-scale Markov decision processes (MDPs) with parameter uncertainty, under the robust MDP paradigm. Previous studies showed that robust MDPs, based on a minimax approach to handle uncertainty, can be solved using dynamic programming for small to medium sized problems. However, due to the "curse of dimen…
A new method for CMDP solving without compromising safety constraints.
problem Solving CMDP problems while adhering to safety constraints.
method Decomposition into reconnaissance and planning MDPs.
result Achieves safe policies for any safety constraint set.
COLA trains linear models on decentralized data without a central coordinator.
problem Learning linear models on decentralized, privacy-sensitive data.
method COLA, a decentralized training algorithm with strong theoretical and practical guarantees.
result Achieves superior communication efficiency, scalability, elasticity, and resilience.
Optimizes learning policies in MDPs with weakly communicating structure.
problem Learning optimal policies in weakly communicating MDPs with generative model.
method Span-based approach, reducing to discounted MDPs for analysis.
result First minimax optimal sample complexity bound for weakly communicating MDPs.
Study shows skewed data labels significantly impact decentralized ML accuracy.
problem Skewed data labels across devices/locations cause significant accuracy loss in decentralized ML.
method Detailed experimental study on skewed data labels, presenting SkewScout system-level approach.
result Skewed data labels are a fundamental challenge for decentralized learning, affecting many applications and models.
New method removes oracle and reduces memory usage for robust MDPs.
problem Applying robust MDPs in practice due to model estimation and oracle requirements.
method Transformed robust MDPs into an alternative form allowing stochastic gradient methods and model-free approach.
result Sample-efficient algorithm with lower storage requirement and no oracle.
New RL method learns to skip states in linearly q π q^π q π -realizable MDPs, simplifying to linear MDPs.
problem Online RL in episodic MDPs with linearly q π q^π q π -realizable action-values. method Derives a novel algorithm that learns to skip states and applies a linear MDP algorithm.
result First polynomial-sample-complexity online RL algorithm for linearly q π q^π q π -realizable MDPs. DeepAveragers solves offline RL by solving derived MDPs from static data.
problem Offline reinforcement learning with limited data.
method Solves derived non-parametric MDPs (DAC-MDPs) using deep representations and costs for under-represented parts.
result The approach can lower-bound performance and scale to complex offline RL problems.
Optimizes learning policies in average-reward MDPs with improved sample complexity.
problem Learning optimal policies in average-reward MDPs with limited samples.
method Reduces to discounted MDPs and uses improved bounds for variance parameters.
result Establishes minimax optimal sample complexity bound of O(SA(H/ε^2))
Enhances parallelism in decentralized learning for larger networks.
problem Scalability limitations in decentralized learning with increasing number of machines.
method Proposes Decentralized Anytime SGD, a novel algorithm that extends parallelism threshold.
result Establishes a theoretical upper bound on parallelism surpassing current state-of-the-art.
BRIDGE improves decentralized learning resilience against Byzantine failures.
problem Decentralized learning in distributed systems is vulnerable to Byzantine failures.
method Introduces BRIDGE, a scalable Byzantine-resilient decentralized gradient descent algorithm.
result Proves algorithmic and statistical convergence guarantees for BRIDGE.
Novel periodic momentum SGD method for decentralized training with linear speedup.
problem Lack of effective momentum schema in decentralized training methods.
method Proposes a novel periodic decentralized momentum SGD method.
result Achieves linear speedup in decentralized training.
Optimistic algorithms achieve logarithmic regret bounds for MDPs without diameter dependence.
problem Achieving logarithmic regret bounds for episodic MDPs without relying on diameter-like quantities.
method Novel 'clipped' regret decomposition applied to optimistic algorithms.
result Smooth interpolation between gap-dependent and minimax rates of convergence.
Single global merging boosts decentralized learning performance.
problem Limited communication in decentralized learning hinders performance.
method Scheduled communication, focusing on final step with global merging.
result Single global merging improves global test performance.
Bayesian algorithms minimize cumulative regret in decentralized multi-agent bandits.
problem Minimizing cumulative regret in a decentralized multi-agent multi-armed bandit problem.
method Proposed decentralized Bayesian multi-armed bandit framework, including Thompson Sampling and Bayes-UCB algorithms.
result Regret scales logarithmically with constants matching those of an optimal centralized agent.
Unified framework for decentralized bilevel optimization with various heterogeneity-correction strategies.
problem Decentralized bilevel optimization with neighborhood communications and data heterogeneity.
method SPARKLE: Single-loop Primal-dual Algorithm for decentralized bilevel optimization, incorporating various heterogeneity-correction techniques.
result Unified convergence analysis for SPARKLE with state-of-the-art convergence rates compared to existing algorithms.
Flexible decentralized MARL framework for cooperative multi-agent learning.
problem Complexity and impracticality of centralized MARL in complicated applications.
method Flexible fully-decentralized actor-critic MARL framework using primal-dual hybrid gradient descent.
result Competitive performance in large-scale cooperative multi-agent environments.
We solve POMDPs by approximating them as finite-state MDPs.
problem Computational challenges in learning optimal policies for POMDPs.
method Transform POMDP into a Superstate MDP, apply TD-learning and policy optimization.
result Finite-time bounds on TD-learning error for non-Markovian dynamics.
New algorithms for Bayesian inference in decentralized learning.
problem Bayesian inference in decentralized learning settings.
method Decentralized SGLD and Decentralized SGHMC.
result Convergence of iterates to target distribution in 2-Wasserstein distance.
Efficiently plans large MDPs with weak function approximations.
problem Planning in large MDPs with limited function approximation capabilities.
method Uses linear value function approximation with weak requirements and a generative oracle.
result Produces almost-optimal actions for any state with polynomial computation time.
This research compiles knowledge on decentralized exchanges with AMM protocols.
problem Improving and developing AMM-based decentralized exchanges.
method Established a general AMM framework, compared mechanics, discussed security and privacy.
result Illustrated conservation and slippage functions of AMM protocols.