This study examines how learning algorithms affect collective action in machine learning.
problem The impact of collective action on machine learning is limited when not considering the choice of learning algorithms.
method Focuses on distributionally robust optimization and stochastic gradient descent, analyzing their effects on collective success.
result The choice of learning algorithm significantly impacts the effective size and success of a collective in machine learning.
Study shows small groups can influence machine learning algorithms.
problem How small groups can influence machine learning algorithms deployed on digital platforms.
method Proposed a theoretical model and conducted experiments on a large-scale language model.
result Small groups can exert significant control over machine learning algorithms.
Algorithm designs neural group actions for symmetric transformations.
problem Designing neural networks for symmetric transformations.
method Develops Neural Group Actions (NGAs) for finite groups, enforcing volume-preserving constraints.
result Demonstrates NGAs for the quaternion group Q8 can learn quantum gate transformations. Recourse explanations can become invalid if collective actions change statistical data.
problem Recourse explanations may become invalid due to collective behavior changing data statistics.
method Formal characterization of conditions under which recourse explanations remain valid under performativity.
result Recourse actions may become invalid if they are influenced by or intervene on non-causal variables.
Collectives can manipulate learning platforms by coordinated data submission, requiring strategic assessments and algorithms.
problem Collectives can influence learning platforms by altering data, posing risks and requiring strategic planning.
method Developed a theoretical and algorithmic framework to understand and mitigate collective manipulation of learning platforms.
result Demonstrated the need for strategic assessments and implementable coordination algorithms to prevent collective manipulation.
Look-ahead reasoning helps predict strategic user behavior on learning platforms.
problem Optimization criteria on learning platforms do not reflect users' priorities.
method Formalized level-k thinking and contrasted collective and selfish behavior.
result Coordination benefits users but does not offer higher-level reasoning advantages in the long run.
Simplifies large action space bandits by selecting representative actions.
problem Efficiently managing large action spaces with correlated outcomes.
method Random sampling and solving of bandit instances to identify representative actions.
result The algorithm selects a smaller set of representative actions that perform nearly as well as the full action space.
In most real-world settings such as recommender systems, finance, and healthcare, collecting useful information is costly and requires an active choice on the part of the decision maker. The decision-maker needs to learn simultaneously what observations to make and what actions to take. This paper incorporates the info…
Paper proposes efficient sample collection strategy for RL.
problem Balancing exploration and exploitation in reinforcement learning.
method Decoupled approach with objective-specific and objective-agnostic strategies.
result Improved or novel sample complexity guarantees for various RL settings.
A new algorithm for cryo-EM data collection that balances reward and latency.
problem Optimizing data collection in cryo-EM experiments with action delays.
method Latency-aware contextual bandit framework and COAF algorithm.
result The COAF algorithm efficiently maximizes reward over time in cryo-EM experiments.
New algorithm learns policies without uniform overlap assumption.
problem Learning optimal policies from non-uniformly collected data.
method Pessimistic Policy Learning (PPL) using lower confidence bounds.
result Efficient policy learning for adaptively collected data.
In this paper we propose a hybrid architecture of actor-critic algorithms for reinforcement learning in parameterized action space, which consists of multiple parallel sub-actor networks to decompose the structured action space into simpler action spaces along with a critic network to guide the training of all sub-acto…
New algorithm for reward-free RL with linear function approximation, reducing sample complexity.
problem Efficiently learning optimal policies without prior reward information in complex environments.
method Developed an algorithm for reward-free RL in linear Markov decision processes, proving sample complexity bounds.
result Polynomial sample complexity in feature dimension and planning horizon, independent of states and actions.
New RL algorithm learns good actions from offline data, reducing uncertainty and divergence.
problem Limited applicability of current RL algorithms in real-world settings due to high costs of exploration.
method Proposes an algorithm for batch RL using a fixed offline dataset, with penalties for policy and value constraints.
result Compared favorably to state-of-the-art methods on 32 continuous-action benchmarks.
In traditional reinforcement learning, an agent maximizes the reward collected during its interaction with the environment by approximating the optimal policy through the estimation of value functions. Typically, given a state s and action a, the corresponding value is the expected discounted sum of rewards. The optima…
Off-policy reinforcement learning aims to leverage experience collected from prior policies for sample-efficient learning. However, in practice, commonly used off-policy approximate dynamic programming methods based on Q-learning and actor-critic methods are highly sensitive to the data distribution, and can make only …
A novel approach learns goal-conditioned policies for locomotion using batch RL.
problem Training goal-conditioned policies for rotation invariant locomotion.
method Data augmentation and Siamese framework for invariance.
result Our approach outperforms existing RL algorithms on 3D locomotion agents.
A new method combines online and offline learning to tackle contextual bandits with missing action support.
problem Learning optimal policies with logged data when the logging policy has deficient support.
method Hybrid approach using online exploration to exploit supported actions and offline learning to avoid unnecessary explorations.
result Determines an optimal policy with theoretical guarantees using minimal online explorations.
The paper classifies circle actions on 6D manifolds with isolated fixed points.
problem Classifying circle actions on 6D manifolds with isolated fixed points.
method Performing equivariant connected sums at fixed points with specific manifolds.
result A sequence of operations can reduce the fixed point data to the empty collection.
In autonomous vehicle (AV) control, allowing mistakes can be quite dangerous and costly in the real world. For this reason we investigate methods of training an AV without allowing the agent to explore and instead having a human explorer collect the data. Supervised learning has been explored for AV control, but it enc…
Optimal policy for multi-armed multi-action bandits with unknown parameters.
problem Optimal sequential action selection for multi-armed multi-action bandits with unknown parameters.
method Occupancy-Measured-Reward Index Policy (OMRIP) and R(MA)^2B-UCB algorithm.
result Asymptotically optimal policy with sub-linear regret and low computational complexity.
We consider a stochastic multi-armed bandit (MAB) problem with delayed impact of actions. In our setting, actions taken in the past impact the arm rewards in the subsequent future. This delayed impact of actions is prevalent in the real world. For example, the capability to pay back a loan for people in a certain socia…
SPEDER extracts state-action abstraction from dynamics for reinforcement learning.
problem Curse of dimensionality and limited applicability of spectral methods.
method Spectral Decomposition Representation (SPEDER) that extracts state-action abstraction from dynamics without policy dependence.
result Theoretical analysis establishes sample efficiency in online and offline settings.
Policy optimization is an effective reinforcement learning approach to solve continuous control tasks. Recent achievements have shown that alternating online and offline optimization is a successful choice for efficient trajectory reuse. However, deciding when to stop optimizing and collect new trajectories is non-triv…
Classically, imitation learning algorithms have been developed for idealized situations, e.g., the demonstrations are often required to be collected in the exact same environment and usually include the demonstrator's actions. Recently, however, the research community has begun to address some of these shortcomings by …
Federated Q-learning achieves linear speedup with heterogeneity, improving sample complexity.
problem Collaborative learning in distributed RL settings with limited data sharing.
method Analyzes synchronous and asynchronous federated Q-learning, proposes importance averaging.
result Achieves linear speedup with heterogeneity, robust to local trajectory heterogeneity.
Unified method for balancing simulation and data collection.
problem Finding optimal solutions with uncertain inputs.
method Bayesian Information Collection and Optimisation (BICO) procedure.
result Algorithm automatically determines optimal balance between optimisation and data collection.
New RL algorithms use lookahead info to maximize rewards.
problem RL in unknown environments without lookahead info.
method Planning using empirical reward and transition distributions.
result Achieves tight regret compared to lookahead info.
In this article we collect a series of observations that constrain actions of many groups on compact manifolds. In particular, we show that "generic" finitely generated groups have no smooth volume preserving actions on compact manifolds while also producing many finitely presented, torsion free groups with the same pr…
New testing method for robust actor-critic bandit algorithms.
problem Balancing data collection for app performance and user adherence.
method Modified actor-critic algorithm and novel testing procedure.
result Testing procedure is robust to critic misspecification.
Action chunking and data exploration improve behavior cloning in robotics.
problem Exponential errors in learning from demonstrations for continuous control tasks.
method Action chunking and exploratory data collection.
result Control-theoretic stability is key to improving imitation learning.
Estimates RL data for dynamic treatment effects using GMM.
problem Estimating dynamic treatment effects from RL data with nonstationary behavior policies.
method Weighted GMM approach to stabilize variance in adaptive RL settings.
result Valid hypothesis testing and confidence regions for dynamic treatment effects.
Researchers identify graph components for unicellular collections.
problem Understanding connected components of surgery graph for unicellular collections.
method Group-theoretic approach involving mapping class group action.
result Connected components enumerated by a homological invariant.
Interactive machine learning improves learning efficiency with user input.
problem Expensive, time-consuming, or risky acquisition of labeled data and decision-making.
method Develops new algorithms for active learning, sequential decision making, and model selection under partial feedback.
result First efficient algorithms achieving exponential label savings and independent of action space size.
In classical reinforcement learning, when exploring an environment, agents accept arbitrary short term loss for long term gain. This is infeasible for safety critical applications, such as robotics, where even a single unsafe action may cause system failure. In this paper, we address the problem of safely exploring fin…
Let Ω=(ωj)j∈I be a collection of pairwise non-isotopic simple closed curves on the closed, orientable, genus g surface Sg, such that ωi and ωj intersect exactly once for i=j. It was recently demonstrated by Malestein, Rivin, and Theran that the cardinality of such a collection is no mo…
We introduce a new class of reinforcement learning methods referred to as {\em episodic multi-armed bandits} (eMAB). In eMAB the learner proceeds in {\em episodes}, each composed of several {\em steps}, in which it chooses an action and observes a feedback signal. Moreover, in each step, it can take a special action, c…
Paper tackles offline RL from mixed datasets with adaptive KL regularizer.
problem Challenges in optimizing RL and BC signals with varying action coverage and multiple action modes.
method Adaptively weighted reverse KL divergence regularizer based on TD3 algorithm.
result Empirically outperforms existing offline RL algorithms in MuJoCo locomotion tasks.
Presents SPEED, an algorithm for optimal policy evaluation in linear bandits with heteroscedastic noise.
problem Optimal data collection for policy evaluation in linear bandits with heteroscedastic reward noise.
method Formulated an optimal design for weighted least squares estimates, derived the optimal sample allocation, introduced SPEED algorithm, and derived regret bounds.
result SPEED leads to policy evaluation with MSE comparable to oracle strategy and significantly lower than random policy execution.
We calculate the cohomology rings of a collection of seven dimensional manifolds supporting an S^3 x S^3-action with one dimensional orbit space. These manifolds are of interest to differential geometers studying non-negative and positive sectional curvature. From this collection, we identify several families of manifo…
Proposes BSI for valid statistical inference on bandit algorithms.
problem Valid statistical inference on bandit algorithms' performance.
method Fits a simulator of the bandit environment from observed data and uses it to estimate mean reward under any policy.
result Proves asymptotically valid confidence intervals and maintains nominal coverage.
Survey on finite group actions on manifolds.
problem Understanding actions of finite groups on manifolds.
method Analyzes the restriction of actions to subgroups and considers group and manifold properties.
result Provides information on subgroup actions and their properties.
In his work on the Farrell-Jones Conjecture, Arthur Bartels introduced the concept of a "finitely F-amenable" group action, where F is a family of subgroups. We show how a finitely F-amenable action of a countable group G on a compact metric space, where the asymptotic dimensions o…
A new algorithm uses IVs to learn optimal policies from observational data.
problem Learning optimal policies from unobserved variable confounded data.
method IV-aided Value Iteration (IVVI) algorithm based on conditional moment restrictions.
result First provably efficient algorithm for instrument-aided offline RL.
Indices of vector fields and 1-forms studied for singular varieties and actions.
problem Understanding indices of vector fields and 1-forms in various contexts.
method Generalization to singular varieties and actions of finite groups.
result New insights into indices of vector fields and 1-forms.
We present a method for a certain class of Markov Decision Processes (MDPs) that can relate the optimal policy back to one or more reward sources in the environment. For a given initial state, without fully computing the value function, q-value function, or the optimal policy the algorithm can determine which rewards w…
Unified stopping rules ensure accurate policies in contextual learning.
problem Stopping data collection to ensure accurate policies in personalized decision problems.
method Developed unified stopping rules based on GLR statistics for pairwise action comparisons.
result Unified stopping rules achieve target precision with fewer samples than benchmarks.
Paper corrects bias in online learning algorithms with endogenous data.
problem Dynamic selection problems in online learning algorithms with endogenous data.
method Instrumental-variable-based algorithm to correct bias, proving central limit theorem.
result Obtains true parameter values and low regret levels.