Weibull weight-scale parameter λ evolves during AdamW training, with alignment, injection, and decay forces driving its growth and relaxation.
problem Understanding the evolution of the Weibull weight-scale parameter λ during AdamW training. method Deriving a leading-order three-force decomposition of the squared weight norm from AdamW updates.
result The alignment force dominates the rise phase, contributing 88-94% of the absolute force budget across four random seeds.
New algorithm shows neural networks can learn without full backpropagation.
problem Stochastic gradient descent with backpropagation is non-biologically plausible.
method Random and fixed backpropagation weights in a feedback alignment algorithm.
result Error converges to zero exponentially fast in overparameterized networks.
DFA trains deep networks by aligning weights then memorizing data.
problem Understanding why DFA works for some networks but not others.
method Two-step learning process: alignment followed by memorization.
result DFA aligns weights to maximize gradient alignment, breaking degeneracy.
B-cos transforms improve neural network interpretability by aligning weights.
problem Improving interpretability of deep neural networks.
method Replacing linear transforms with B-cos transforms that promote weight-input alignment during training.
result B-cos transforms lead to highly interpretable and task-relevant linear summaries of neural network computations.
Proposes neuron alignment to optimize mode connectivity in neural networks.
problem Understanding and optimizing mode connectivity in deep neural networks.
method Introduces neuron alignment to approximate optimal weight permutations and improve mode connectivity.
result Neuron alignment significantly alleviates robust loss barriers and improves model robustness and accuracy.
Improves inference-time alignment for diffusion models without updating weights.
problem Aligning diffusion models without updating weights for high-reward outputs.
method Trust-Region Iterative Twisted Sequential Monte Carlo (TRI-TSMC) for variance reduction and efficiency.
result Improves primary alignment objectives on text generation tasks.
Novel alignment framework for text-to-image generation using diffusion models and flow matching.
problem Improving text-to-image generation with minimal computational resources.
method Proposes a novel alignment framework that decomposes the score function into pre-trained score plus a conditional expectation of the reward.
result Achieves comparable performance to finetuning-based models with reduced computational cost.
SONA improves conditional generation by balancing authenticity and alignment.
problem Challenges in balancing authenticity and conditional alignment in conditional generative models.
method SONA integrates unconditional discrimination, matching-aware supervision, and adaptive weighting to balance authenticity and alignment.
result SONA achieves superior sample quality and conditional alignment compared to state-of-the-art methods.
Transformer models align words through attention weights, closely approximating Optimal Transport.
problem Understanding the internal mechanism of transformer models in language processing.
method Empirical evidence and theoretical analysis of attention weights and their relation to Optimal Transport.
result Transformer models can simulate gradient descent on the dual of entropy-regularized OT problem, providing a theoretical foundation for token alignment.
Study connects database alignment and planted matching using Gaussian features.
problem Identify matching between correlated user features in anonymized databases.
method Derived results for database alignment and planted matching, showing connections and thresholds.
result Performance thresholds for database alignment converge to planted matching when feature dimensionality is sufficiently high.
New learning rules for wide neural networks without backpropagation.
problem Training wide neural networks efficiently and without backpropagation.
method Input-weight alignment driven by gradient descent in the NTK regime.
result Biologically-motivated learning rules equivalent to backpropagation in wide networks.
This paper establishes risk convergence and asymptotic weight matrix alignment --- a form of implicit regularization --- of gradient flow and gradient descent when applied to deep linear networks on linearly separable data. In more detail, for gradient flow applied to strictly decreasing loss functions (with similar re…
Sharp threshold found for aligning Gaussian-weighted graphs.
problem Reconstructing planted permutations in Gaussian-weighted graphs.
method Analysis of MAP estimator and second moment method.
result Sharp information-theoretic threshold for exact recovery.
Current algorithms for deep learning probably cannot run in the brain because they rely on weight transport, where forward-path neurons transmit their synaptic weights to a feedback path, in a way that is likely impossible biologically. An algorithm called feedback alignment achieves deep learning without weight transp…
Ongoing studies have identified similarities between neural representations in biological networks and in deep artificial neural networks. This has led to renewed interest in developing analogies between the backpropagation learning algorithm used to train artificial networks and the synaptic plasticity rules operative…
STRAPSim measures ETF portfolio similarity better than existing methods.
problem Measuring portfolio similarity for ETFs and portfolios.
method Semantic, two-level, residual-aware portfolio similarity computation.
result STRAPSim outperforms existing methods in predictive accuracy and ranking alignment.
A new method learns continuous guidance weights to improve diffusion model quality and distributional alignment.
problem Improving perceptual quality and distributional alignment of samples from conditional diffusion models.
method Learned continuous guidance weights ωc,(s,t) are used to minimize distributional mismatch and reward guided sampling. result Improvements in Fréchet inception distance (FID) for image generation and better image-prompt alignment in text-to-image applications.
A new flow method solves the weighted Yamabe problem with boundary.
problem Solving the weighted Yamabe problem on metric measure spaces with boundary.
method Introduced a Yamabe-type flow with a specific geometric setup.
result Long-time existence and convergence of the flow proved.
HSACC improves multi-view clustering of incomplete data.
problem Challenges in clustering incomplete multi-view data.
method Hierarchical Semantic Alignment and Cooperative Completion framework.
result HSACC outperforms state-of-the-art methods on benchmark datasets.
NTK-SAP improves neural network pruning by aligning training dynamics.
problem Improving neural network pruning to reduce training time and memory.
method Prune connections based on the spectrum of the Neural Tangent Kernel (NTK), using multiple random weight realizations and random inputs.
result Empirically, NTK-SAP achieves better performance than all baselines on multiple datasets.
Gradient descent aligns neural feature matrices with pre-activation tangent features.
problem Understanding neural feature learning mechanisms.
method Analytical proof of alignment between weight matrices and pre-activation tangent features.
result Derivative alignment occurs almost surely in high-dimensional settings.
We analyze deep neural networks using convex duality to reveal hidden layer structures.
problem Understanding the structure of deep neural networks.
method Introducing a convex analytic framework to characterize hidden layer weights.
result Optimal hidden layer weights align with previous layers via duality.
Study improves understanding and performance of FA learning rules in neural networks.
problem Lack of theoretical understanding and limited applications of Feedback Alignment (FA) methods.
method Introduces a unified framework linking synaptic weight changes to implicit regularization, providing convergence conditions and empirical evidence.
result Better alignment can enhance FA performance on complex multi-class tasks.
New bounds for PDA using partial optimal transport improve domain alignment.
problem Scarcity of labeled target data with abundant source data.
method Derive theoretical bounds based on partial optimal transport.
result Theoretical bounds support partial Wasserstein distance for domain alignment.
Adversarial training makes logistic regression weight loss landscapes sharper.
problem Understanding why adversarial training sharpens the weight loss landscape in logistic regression.
method Theoretical analysis of linear logistic regression model with L2 norm constraints, and experiments on ResNet18.
result Adversarial training sharpens the weight loss landscape in linear logistic regression models.
Decision-alignment evaluates uncertainty quantification for decision-relevant UQ
problem Evaluation of uncertainty quantification metrics
method Introduce decision-alignment
result Proper scoring rules align with decision utility
Let (MN,g,e−fdv) be a complete smooth metric measure space with ∞-Bakry-Émery Ricci tensor bounded from below. We derive elliptic gradient estimates for positive solutions of a weighted nonlinear parabolic equation \begin{align*} \displaystyle \Big(Δ_f - \frac{\partial}{\partial t}\Big) u(x,t) +q(x,t)u^α…
QAlign improves language model alignment with less compute, outperforming existing methods.
problem Improving language model performance with limited test-time computation.
method QAlign: sampling from optimal aligned distribution using Markov chain Monte Carlo.
result Consistent improvements over existing methods on various benchmarks.
Network alignment is a critical task to a wide variety of fields. Many existing works leverage on representation learning to accomplish this task without eliminating domain representation bias induced by domain-dependent features, which yield inferior alignment performance. This paper proposes a unified deep architectu…
Regularization improves stability and consistency of sparse autoencoders.
problem Varying features across random seeds and training choices in SAEs.
method Added L1 or L2 penalties on encoder and decoder weights.
result L2 regularization increases cross-seed feature consistency.
A new method aligns source and target distributions by tuning their weights.
problem Domain adaptation on unlabeled target datasets using labeled source datasets.
method Weighted Joint Distribution Optimal Transport (WJDOT) method that finds alignment between source and target distributions and re-weighting of source distributions.
result Achieves state-of-the-art performance on simulated and real-life datasets.
Artificial neural networks are most commonly trained with the back-propagation algorithm, where the gradient for learning is provided by back-propagating the error, layer by layer, from the output layer to the hidden layers. A recently discovered method called feedback-alignment shows that the weights used for propagat…
One-step Bellman alignment improves online RL by reducing task mismatch.
problem Online RL struggles with task similarity defined by rewards or transitions.
method One-step Bellman alignment and re-weighted targeting (RWT) to correct task mismatch.
result Regret bounds show task shift complexity, not target MDP, affects performance.
This paper extends neural collapse to regression problems, revealing key features and structures.
problem Understanding the structure learned by deep neural networks in regression tasks.
method Established Neural Regression Collapse (NRC) across different models, analyzing feature and weight alignments.
result Deep neural regression models exhibit a collapsed feature space, aligning with target dimensions and covariances.
Optimizes pruning masks for neural networks using probabilistic fine-tuning and PAC-Bayes bounds.
problem Improving neural network performance through adaptive pruning of weights.
method Optimizes stochastic pruning masks by minimizing expected loss, considering data-adaptive regularization and feature alignment.
result Probabilistic fine-tuning leads to improved test error over baseline methods in neural networks.
A new method for drone-based geo-localization using style and spatial alignment.
problem Geo-localization of drone-view images with satellite-view images using pre-annotated GPS tags.
method Orientation-based method to align patterns, new branch to extract aligned partial features, style alignment strategy.
result The proposed method outperforms state-of-the-art alternatives in geo-localization accuracy.
CoDA Nets improve interpretability in neural networks.
problem Improving interpretability in neural networks.
method Dynamic Alignment Units (DAUs) for input-dependent linear transformations.
result CoDA Nets achieve on par results with ResNet and VGG models on complex datasets.
Recently theoretical guarantees have been obtained for matrix completion in the non-uniform sampling regime. In particular, if the sampling distribution aligns with the underlying matrix's leverage scores, then with high probability nuclear norm minimization will exactly recover the low rank matrix. In this article, we…
Nowadays, multi-view clustering has attracted more and more attention. To date, almost all the previous studies assume that views are complete. However, in reality, it is often the case that each view may contain some missing instances. Such incompleteness makes it impossible to directly use traditional multi-view clus…
Recent advances in deep neural networks (DNNs) owe their success to training algorithms that use backpropagation and gradient-descent. Backpropagation, while highly effective on von Neumann architectures, becomes inefficient when scaling to large networks. Commonly referred to as the weight transport problem, each neur…
Proposes DWMD for better matching of hidden representations across domains.
problem Measuring data distribution discrepancy between semantically related domains for feature representation matching.
method DWMD, a moment-based probability distribution metric that explicitly orders and weights higher-order moments.
result DWMD is error-free and can strictly reflect distribution differences without feature distribution assumptions.
Using back-propagation and its variants to train deep networks is often problematic for new users. Issues such as exploding gradients, vanishing gradients, and high sensitivity to weight initialization strategies often make networks difficult to train, especially when users are experimenting with new architectures. Her…
FedAVOT improves federated learning by aligning user distributions.
problem Partial client participation leads to biased and unstable updates in federated learning.
method Formulates aggregation as masked optimal transport to align availability and importance distributions.
result Achieves a standard O(1/√T) rate, independent of the number of participating users per round.
New CV method reduces bias in spatial prediction models.
problem Bias in standard cross-validation due to uneven sampling.
method Target-Weighted Cross-Validation (TWCV) framework.
result Weighted CV approaches reduce bias in prediction error.
The paper reveals low-rank structure in neural network gradients, influenced by data and model parameters.
problem Investigating low-rank structure in gradients of neural networks under relaxed assumptions.
method Spiked data model, relaxation of isotropy assumptions, analysis of mean-field and neural-tangent-kernel scalings.
result Gradient of input weights is approximately low rank, dominated by two rank-one terms.
The paper approaches the task of handwritten text recognition (HTR) with attentional encoder-decoder networks trained on sequences of characters, rather than words. We experiment on lines of text from popular handwriting datasets and compare different activation functions for the attention mechanism used for aligning i…
Propose an XMSE-aware mixed estimator for EB that interpolates between ML and EB shrinkage.
problem Kernel-based EB estimation may be worse than ML when the kernel is poorly aligned with the true parameter.
method An XMSE-aware mixed estimator that interpolates between ML and EB shrinkage.
result Fixed-weight XMSE is a scalar quadratic, yielding a closed-form oracle mixing weight that is no worse than both ML and the base EB estimator at the XMSE scale.
Optimal transport aligns source and target distributions for domain adaptation.
problem Unsupervised domain adaptation with joint class-conditional and label shifts.
method Minimizes importance weighted loss and Wasserstein distance for aligned marginals and class-conditional distributions.
result Our method outperforms competitors on various domain adaptation tasks.