Direct feedback alignment reduces data movement in neural networks.
problem Efficiency and energy-efficiency in training large neural networks.
method Sparse feedback matrix for local learning, reducing data movement and compute.
result Orders of magnitude improvement in data movement and 2x improvement in multiply-and-accumulate operations.
Direct Feedback Alignment performs well on diverse deep learning tasks and architectures.
problem The limitations of backpropagation in parallelizing and scaling to modern deep learning tasks.
method Direct Feedback Alignment approach applied to neural view synthesis, recommender systems, geometric learning, and natural language processing.
result Direct Feedback Alignment successfully trains a wide range of state-of-the-art deep learning architectures with performance close to fine-tuned backpropagation.
New training method improves neural network performance.
problem Improving neural network performance with direct feedback alignment.
method Direct feedback alignment with best practices and observations.
result Characterization of bottleneck effect in narrow layers.
Photonic co-processor speeds up training of large neural networks.
problem Training large neural networks with backpropagation is inefficient and communication is a bottleneck.
method Direct Feedback Alignment (DFA) with a photonic accelerator.
result Photonic accelerator can compute random projections with trillions of parameters.
Improved CNN training with BDFA reduces computational cost and improves accuracy.
problem Low training performance of DFA in CNN.
method Combining DFA with BP, introducing feedback weight initialization, and proposing BDFA.
result BDFA shows better performance than conventional BP, especially in small datasets.
The abstract explores connections between reinforcement learning, scaling, and diffusion.
problem Aligning reinforcement learning with human feedback and scaling techniques.
method Clarifying connections between reinforcement learning, scaling, and diffusion.
result Introducing a resampling approach for alignment and reward-directed diffusion models.
Develops methods to correct bias in AI feedback for more accurate alignment.
problem Systematic bias in AI feedback compared to human labels.
method Two debiased alignment methods: DDPO and DIPO.
result Methods improve alignment efficiency and performance close to human-labeled data.
Efficiently identifies good policies by choosing contexts for human feedback.
problem Efficiently acquiring human feedback for preference alignment in large language models.
method Formalizes active exploration as a dueling bandit problem and proposes an active exploration algorithm with a polynomial worst-case regret bound.
result Proposed method outperforms baselines with limited human preferences on various language models and datasets.
Study shows Direct Feedback Alignment fails to offer more efficient scaling than backpropagation.
problem Understanding and optimizing training methods for neural networks.
method Use of scaling laws to compare Direct Feedback Alignment (DFA) and backpropagation.
result DFA fails to offer more efficient scaling than backpropagation.
DFA trains deep networks by aligning weights then memorizing data.
problem Understanding why DFA works for some networks but not others.
method Two-step learning process: alignment followed by memorization.
result DFA aligns weights to maximize gradient alignment, breaking degeneracy.
This paper introduces f f f -DPO, a generalized approach to Direct Preference Optimization using diverse divergence constraints.
problem Aligning large language models with human preferences while mitigating safety risks.
method Incorporates diverse divergence constraints to simplify the relationship between reward and optimal policy, eliminating the need for estimating the normalizing constant.
result Optimizes LLMs to align with human preferences more efficiently and under a broader set of divergence constraints.
RLHF uses human feedback to train AI models, posing statistical challenges.
problem Aligning AI models with human preferences using noisy, subjective feedback.
method Supervised fine-tuning, reward modeling, policy optimization, statistical ideas.
result Statistical methods for reward function learning and policy optimization.
Artificial neural networks are most commonly trained with the back-propagation algorithm, where the gradient for learning is provided by back-propagating the error, layer by layer, from the output layer to the hidden layers. A recently discovered method called feedback-alignment shows that the weights used for propagat…
Paper addresses RLHF alignment challenges with novel algorithms.
problem Challenges in RLHF alignment, especially in strategic exploration.
method Develops a reverse-KL regularized contextual bandit formulation and proposes efficient algorithms with theoretical guarantees.
result Proposed methods significantly outperform existing RLHF algorithms in real-world experiments.
Hölder-DPO aligns models robustly with noisy human feedback.
problem No existing alignment methods can handle severe label noise.
method Proposes Hölder-DPO, a principled alignment loss with provable redescending property.
result Hölder-DPO enables scalable human feedback valuation and improves model alignment.
ADPO optimizes relative advantage in reinforcement learning from human feedback.
problem Optimizing policy alignment in reinforcement learning from human preferences.
method ADPO explicitly parameterizes the optimal structure through anchored logits, decoupling response quality from prior popularity.
result Empirically, ADPO achieves state-of-the-art performance on reasoning tasks, outperforming GRPO by 30.9 percent.
PILAF optimizes reward models from human feedback for better policy alignment.
problem Creating accurate reward models from human feedback for policy optimization.
method Policy-Interpolated Learning for Aligned Feedback (PILAF) that explicitly aligns preference learning with maximizing underlying oracle reward.
result PILAF is optimal from both optimization and statistical perspectives, demonstrating strong performance in RLHF settings.
DPA aligns LLMs with multi-objective rewards for diverse user preferences.
problem Fine-grained control over LLMs for diverse user needs.
method Integrates multi-objective reward modeling and directional preference control.
result DPA offers better performance trade-offs and intuitive user control over LLM generation.
Study improves understanding and performance of FA learning rules in neural networks.
problem Lack of theoretical understanding and limited applications of Feedback Alignment (FA) methods.
method Introduces a unified framework linking synaptic weight changes to implicit regularization, providing convergence conditions and empirical evidence.
result Better alignment can enhance FA performance on complex multi-class tasks.
New framework formalizes RLHF trilemma: improving safety, fairness, and robustness is computationally infeasible.
problem Aligning large language models with diverse human values while maintaining computational feasibility and robustness.
method Complexity-theoretic analysis integrating statistical learning theory and robust optimization.
result Achieving both representativeness (epsilon <= 0.01) and robustness (delta <= 0.001) for global-scale populations requires super-polynomial operations.
LLM trading agents show risk feedback can improve alignment without fine-tuning.
problem Aligning LLM trading agents with financial risk.
method TradeArena testbed, risk reports, execution simulation, memory replay.
result Risk feedback can improve alignment without fine-tuning, but not universally.
New method learns interpretable concepts from user feedback for high-dimensional data.
problem Lack of interpretable concepts in machine learning models trained on high-dimensional tabular data.
method Proposes a method for learning transparent concept definitions from user labeling of concept features, not instances.
result Demonstrates more efficient learning of aligned concept definitions from user feedback compared to alternative transparent approaches.
New algorithms avoid weight transport, outperforming current deep learning methods.
problem Current deep learning algorithms rely on weight transport, which is biologically implausible.
method Two mechanisms: weight mirror and modified Kolen-Pollack algorithm, using random feedback weights.
result These mechanisms outperform feedback alignment and other methods on visual recognition tasks.
SAIL improves online alignment of large language models with minimal feedback.
problem Offline RLHF methods often lead to sub-optimal performance due to fixed preference datasets.
method SAIL uses bilevel optimization and a single-level first-order method to iteratively refine model alignment.
result SAIL significantly improves alignment performance on open-sourced datasets with minimal computational overhead.
OOM-RL uses financial market losses to align AI agents in autonomous systems.
problem Constrained alignment of autonomous software agents in live financial markets.
method Deploying agents in live financial markets to enforce strict test-driven workflows.
result Final OOM-RL-aligned system achieved a stable equilibrium with an annualized Sharpe ratio of 2.06.
Improved text-to-image alignment using iterative VQA feedback.
problem Misalignment between text prompts and generated images, especially for complex inputs.
method Decompose complex prompts into assertions, evaluate each using VQA, combine scores iteratively.
result Significantly higher correlation with human ratings compared to CLIP, BLIP scores.
Paper proposes efficient RLHF methods for LLMs using active queries.
problem Efficiently aligning LLMs with human preferences using RLHF.
method Formalizes RLHF as a dueling bandit problem, introduces APPO and ADPO algorithms.
result ADPO achieves similar performance to state-of-the-art methods with fewer queries.
The paper improves DFA for CNN and RNN training to match BP accuracy.
problem Low accuracy in CNN and RNN training using DFA.
method Divide network into modules, apply DFA within, use sparse backward weight, and incorporate dilated convolution and sparse matrix multiplication.
result Achieves BP-level accuracy in CNN and RNN training.
Unified framework simplifies DPO algorithms for LLM alignment.
problem Vast number of DPO variants complicates model alignment.
method Mutual information inspired unifying framework with flexible priors.
result Many DPO variants can be derived from the new framework.
Improved DPO framework penalizes preference uncertainty to avoid overoptimization.
problem Aligning LLMs to human preferences is challenging due to varied, context-dependent, and ambiguous preferences.
method Developed a pessimistic framework for DPO by introducing preference uncertainty penalization schemes.
result Improved overall performance and better completions on high-uncertainty responses compared to vanilla DPO.
FA algorithm provides convergence guarantees for deep linear networks.
problem Training efficiency and convergence of deep neural networks.
method Theoretical analysis of Feedback Alignment (FA) algorithm for deep linear networks.
result Certain initializations lead to implicit anti-regularization, affecting learning effectiveness.
New method prevents RLHF alignment collapse by accounting for policy's influence on reward model updates.
problem Iterative RLHF leads to alignment collapse where policies exploit RM's blind spots.
method Foresighted policy optimization (FPO) restores missing steering term via regularization.
result FPO prevents alignment collapse on LLM alignment pipelines using Llama-3.2-1B.
Unified framework for aligning LLMs from human feedback.
problem Lack of strong theoretical justification for RLHF and inability to compare methods.
method Reframed alignment as distribution learning from pairwise preferences, proposing three principled objectives.
result Proposed objectives achieve strong non-asymptotic convergence to target LM.
A new method steers Gaussian distributions with minimal effort.
problem Steering high-dimensional Gaussian distributions efficiently.
method Sliced feedback controller using one-dimensional projections and averaging.
result The method steers Gaussian distributions to targets efficiently.
Method tackles uncertainty in reward models for LLMs from heterogeneous human feedback.
problem Uncertainty in reward models for LLMs from heterogeneous human feedback.
method Heterogeneous preference framework and alternating gradient descent algorithm.
result Established theoretical guarantees for estimator convergence and asymptotic distribution.
New framework transfers latent knowledge from weak to strong models.
problem Aligning superhuman LLMs with human feedback.
method Transfer learning framework using refinement approach.
result Proves weak-to-strong generalization is possible.
RAFT fine-tunes models using high-quality samples to align them with human preferences.
problem Aligning generative models with human ethics and preferences.
method RAFT selects high-quality samples, discards undesired behavior, and fine-tunes the model on filtered samples.
result RAFT improves model performance in reward learning and automated metrics.
Enhances AI models with human feedback for noisy data.
problem Improving AI model alignment with human feedback in noisy environments.
method Two-stage SL+LHF framework connecting machine learning with human feedback.
result The LNCA ratio identifies conditions for SL+LHF superiority over pure SL.
Paper addresses online alignment of large language models under uncertain preference feedback.
problem Online alignment of large language models with misspecified preference feedback.
method Formulates an oracle-robust objective as a worst-case optimization problem for log-linear policies, and develops projected stochastic composite updates.
result Shows that the robust objective admits an exact closed-form decomposition and achieves O ~ ( ε − 2 ) \widetilde{O}(\varepsilon^{-2}) O ( ε − 2 ) oracle complexity. Dual active learning improves RLHF by selecting optimal conversations and teachers.
problem Efficiently aligning LLMs with human preferences using RLHF from feedback.
method Offline RL for conversation and teacher selection, dual active reward learning, pessimistic RL.
result The proposed algorithm achieves minimal generalized variance and outperforms state-of-the-arts.
New algorithm shows neural networks can learn without full backpropagation.
problem Stochastic gradient descent with backpropagation is non-biologically plausible.
method Random and fixed backpropagation weights in a feedback alignment algorithm.
result Error converges to zero exponentially fast in overparameterized networks.
Proposes a robust algorithm for aligning large language models with human preferences.
problem Misspecification in preference models, reference policies, and reward functions.
method Doubly robust preference optimization algorithm.
result Superior and more robust performance compared to state-of-the-art algorithms.
A new method reduces the computational burden of safety alignment for large language models.
problem Safety concerns in large language models and the need to align them with human preferences.
method Optimal dualization approach to reduce constrained alignment to an unconstrained problem.
result Our algorithms MoCAN and PeCAN significantly reduce computational burden and improve training stability.
The paper proposes a new method for learning reward models from ordinal feedback, improving upon binary feedback.
problem Learning reward models from human preferences using binary feedback discards useful samples and loses fine-grained information.
method The paper introduces a framework for learning reward models under ordinal feedback, generalizing the Bradley-Terry model.
result Ordinal feedback reduces the Rademacher complexity compared to binary feedback, leading to better reward learning.
Analysis of deep neural networks under various learning rules reveals dynamics of feature and prediction learning.
problem Understanding how different learning rules affect feature and prediction dynamics in deep neural networks.
method Analysis of infinite-width deep networks trained with gradient descent and various learning rules.
result The evolution of the output function is governed by an effective neural tangent kernel (eNTK), which varies depending on the learning rule and training regime.
Biologically plausible learning algorithms can match BP on large datasets.
problem Learning algorithms that are biologically plausible often perform poorly on large datasets.
method Evaluation of sign-symmetry and feedback alignment algorithms on ImageNet and MS COCO.
result Sign-symmetry algorithm can match BP performance on ImageNet and MS COCO.
New method constructs tilings of the plane using directed edges and alignments.
problem Modeling tilings of the Euclidean or hyperbolic plane as presheaves over categories.
method Introducing finite categories for polygons with labeled directed edges, constructing reflective alignments.
result Characterizing alignments of tilings by comparing edge directions and generating families with elegant symmetry.
SLHF uses sequential game theory to optimize preferences from human feedback.
problem Optimizing preferences from human feedback in sequential settings.
method SLHF frames the problem as a sequential-move game between Leader and Follower, decomposing the optimization into refinement and adversarial optimization.
result SLHF achieves strong alignment across diverse preference datasets and scales to large models.