A new method reduces memory usage in neural networks by 36-81%.
problem Minimizing memory usage in neural networks during backpropagation.
method Formalized as a graph theory problem, uses dynamic programming to minimize computational overhead.
result Reduces peak memory consumption by 36-81% on various networks.
A new algorithm reduces memory usage for deep learning models.
problem Training deep learning models requires significant memory.
method Dynamic Tensor Rematerialization (DTR) is a greedy online algorithm that dynamically plans recomputations.
result DTR achieves comparable performance to optimal static checkpointing with only a small memory budget.
The Long-Short-Term-Memory Recurrent Neural Networks (LSTM RNNs) are a popular class of machine learning models for analyzing sequential data. Their training on modern GPUs, however, is limited by the GPU memory capacity. Our profiling results of the LSTM RNN-based Neural Machine Translation (NMT) model reveal that fea…
ACP-UCB1 ranks arms based on upper-tail performance, improving stochastic bandit algorithms.
problem Stochastic bandit algorithms often favor arms with strong upper-tail performance, which is not well-addressed by classical mean-reward criteria.
method ACP-UCB1 combines an adaptive conformal estimate of the upper endpoint with a UCB-type optimism bonus.
result ACP-UCB1 achieves logarithmic upper-quantile regret with per-arm contribution \(O(
icefrac{\log n}{Δ_j^{\mathrm{ACP}}})\).
Bieri, Geoghegan and Kochloukova computed the BNSR-invariants Σm(F) of Thompson's group F for all m. We recompute these using entirely geometric techniques, making use of the Stein--Farley CAT(0) cube complex on which F acts.
An iterative method of learning has become a paradigm for training deep convolutional neural networks (DCNN). However, utilizing a non-iterative learning strategy can accelerate the training process of the DCNN and surprisingly such approach has been rarely explored by the deep learning (DL) community. It motivates thi…
New ODE solvers improve training efficiency and accuracy.
problem Training Neural ODEs requires efficient and accurate gradient calculation.
method Presented algebraically reversible ODE solvers that are time and memory efficient, calculate exact gradients, and are numerically stable.
result Reversible solvers strictly improve upon previous architectures in efficiency and accuracy.
We investigate the integration of a planning mechanism into sequence-to-sequence models using attention. We develop a model which can plan ahead in the future when it computes its alignments between input and output sequences, constructing a matrix of proposed future alignments and a commitment vector that governs whet…
This project combines recent advances in experience replay techniques, namely, Combined Experience Replay (CER), Prioritized Experience Replay (PER), and Hindsight Experience Replay (HER). We show the results of combinations of these techniques with DDPG and DQN methods. CER always adds the most recent experience to th…
This paper improves hierarchical community detection efficiency using local structural properties.
problem Efficiency of hierarchical community detection methods in large networks.
method Use of local structural network properties as proxies to improve efficiency.
result Achieves competitive results in modularity with improved efficiency.
We discuss investment allocation to multiple alpha streams traded on the same execution platform with internal crossing of trades and point out differences with allocating investment when alpha streams are traded on separate execution platforms with no crossing. First, in the latter case allocation weights are non-nega…
It is the main goal of this paper to propose a novel method to perform matrix completion on-line. Motivated by a wide variety of applications, ranging from the design of recommender systems to sensor network localization through seismic data reconstruction, we consider the matrix completion problem when entries of the …
Variance reduced stochastic gradient (SGD) methods converge significantly faster than the vanilla SGD counterpart. However, these methods are not very practical on large scale problems, as they either i) require frequent passes over the full data to recompute gradients---without making any progress during this time (li…
Unified approach to multiclass classification using Gabriel graphs.
problem Improving multiclass classification accuracy and efficiency.
method Integrates Gabriel graphs for binary and multiclass classification, proposing new activation functions and support edge neurons.
result Experimental results show superior performance compared to previous GG-based classifiers.
Online learning algorithms require to often recompute least squares regression estimates of parameters. We study improving the computational complexity of such algorithms by using stochastic gradient descent (SGD) type schemes in place of classic regression solvers. We show that SGD schemes efficiently track the true s…
A new metric mav offers a practical alternative to costly Riemannian distance.
problem Efficiently compute Riemannian distance on SE(3) invariant metrics.
method Propose mav distance, defined as Riemannian length of a curve.
result Mav distance offers a trainable invariant for geometric deep learning.
We propose different schemes for option hedging when asset returns are modeled using a general class of GARCH models. More specifically, we implement local risk minimization and a minimum variance hedge approximation based on an extended Girsanov principle that generalizes Duan's (1995) delta hedge. Since the minimal m…
Recurrent neural networks (RNNs) provide state-of-the-art performance in processing sequential data but are memory intensive to train, limiting the flexibility of RNN models which can be trained. Reversible RNNs---RNNs for which the hidden-to-hidden transition can be reversed---offer a path to reduce the memory require…
Sparse Transformers reduce memory and time requirements for long sequence modeling.
problem Quadratic growth in memory and time with transformer sequence length.
method Sparse factorizations of attention matrix, deeper network training, attention matrix recomputation, fast attention kernels.
result Sparse Transformers can model sequences up to tens of thousands of timesteps with hundreds of layers.
ERNet reduces CNN bandwidth needs for imaging tasks.
problem High DRAM bandwidth requirements for CNNs in imaging.
method ERNet uses block-based inference and temporary layer expansion.
result ERNet outperforms state-of-the-art models in image denoising and super-resolution.
Gradient descent, or negative gradient flow, is a standard technique in optimization to find minima of functions. Many implementations of gradient descent rely on discretized versions, i.e., moving in the gradient direction for a set step size, recomputing the gradient, and continuing. In this paper, we present an appr…
Active subspaces on Riemannian manifolds generalize Euclidean principles.
problem Understanding how scalar-valued quantities change over Riemannian manifolds.
method Generalization of active subspaces from Euclidean to Riemannian spaces using parallel transport.
result The method provides a new way to study scalar-valued quantities on manifolds, differing from extrinsic approaches.
We introduce a methodology to construct parsimonious probabilistic models. This method makes use of Information Filtering Networks to produce a robust estimate of the global sparse inverse covariance from a simple sum of local inverse covariances computed on small sub-parts of the network. Being based on local and low-…
iCVI-ARTMAP accelerates clustering with adaptive resonance theory and validity indices.
problem Improving clustering efficiency and accuracy using adaptive resonance theory.
method Integrates adaptive resonance theory (ARTMAP) with incremental cluster validity indices (iCVIs) for clustering.
result Significantly reduces clustering time and outperforms other methods on synthetic and real-world data.
UAVs learn to collect data from IoT sensors efficiently.
problem Efficient UAV path planning for wireless data collection.
method Deep reinforcement learning (DDQN) with experience replay and convolutional layers.
result UAV control policy generalizes over changing scenario parameters.
SON-GOKU uses graph coloring to improve multi-task learning by partitioning tasks into compatible groups.
problem Gradient interference between conflicting multi-task learning objectives slows convergence and model performance.
method SON-GOKU computes gradient interference, constructs an interference graph, and applies greedy graph-coloring to partition tasks.
result SON-GOKU consistently outperforms baselines and state-of-the-art multi-task optimizers on six datasets.
MRC improves credit assignment in multi-agent LLM systems, achieving high returns and transparency.
problem Lack of principled credit assignment in multi-agent LLM decision systems, vulnerability to regime shifts, and limited transparency.
method Market Regime Council (MRC) computes exact Shapley credits, uses exponentially weighted performance histories, Bayesian adaptive mixture, and regime-dependent multipliers.
result MRC achieves a Sharpe ratio of 1.51 and a cumulative return of 440.1% over 1,037 trading days, ranking first on CR, SR, and IR.
Framework for analyzing dynamic topological changes in point clouds using persistent homology and dynamic optimal transport.
problem Analyzing transient structural reorganizations during dynamic phase transitions in time-evolutionary point clouds.
method Hierarchical dynamic evaluation framework driven by topological and hypergraph reconstruction strategy.
result Combining transport-based alignment with multi-scale entropy diagnostics for dynamic topological analysis.
Partial soft-matching distance improves neural representation comparison by allowing some neurons to remain unmatched.
problem Neural representations are noisy and contain outliers, making traditional matching methods unreliable.
method Extends soft-matching distance to a partial optimal transport setting, allowing some neurons to remain unmatched.
result Partial soft-matching provides robust correspondences that are more reliable under noise and outliers.
This paper interpolates reward functions to predict optimal value functions in MORL.
problem Finding optimal value functions in MORL requires recomputing for each set of weights.
method Interpolating reward function weights to smooth value function transformations.
result Smooth interpolation of optimal value functions over reward function weights.
Neural networks combining multiple data sources can reverse preferences, affecting decision reliability.
problem Preference reversals in neural networks under pooled data.
method Formalized through Case-Based Decision Theory, analyzed Gram geometry, introduced regularization, and developed auditing methods.
result Pooled refitting can reverse shared preferences, and conditions for preserving preferences are derived.
Enhances data valuation by integrating global and local statistical properties.
problem Insufficient consideration of global and local statistical properties in data valuation methods.
method Proposes a method that fuses global and local statistical properties into regularization terms for Shapley value estimation and dynamic data valuation.
result Demonstrates improved performance and efficiency of data valuation methods through integration of global and local statistical properties.
A new concordance loss improves model performance and reliability in survival prediction.
problem Inconsistent evaluation of deep survival models using likelihood losses.
method Proposed a value-monotone concordance loss (SCL) to improve reliability and optimization.
result SCL achieves comparable discrimination and is the best or within one standard deviation of the best C-index across multiple datasets.
CLQT benchmarks LLM portfolio managers by evaluating their decision-making process, not just returns.
problem Most benchmarks rank LLMs by returns, ignoring their decision-making process and potential for look-ahead leakage.
method CLQT reframes evaluation as diagnosis, using a closed-loop, cost-aware, strategy-consistent environment with a five-stage cycle.
result CLQT provides a durable map of agent competencies and limitations, separating outcome from process.
Memory-efficient optimizers fail to track a subspace, leading to unpredictable model performance.
problem Memory-efficient optimizers fail to track a subspace, leading to unpredictable model performance.
method Analyzing the behavior of memory-efficient optimizers like GaLore, which project gradients onto a rank-r subspace recomputed every T steps.
result Memory-efficient optimizers fail to track a subspace, leading to unpredictable model performance.
Examines algorithmic modeling across three cultures.
problem Tackles algorithmic modeling in different cultural contexts.
method Uses parametric regressions, interpretable algorithms, and complex algorithms.
result Extension of Leo Breiman's thesis to include cultural differences.
Playing repeated matrix games (RMG) while maximizing the cumulative returns is a basic method to evaluate multi-agent learning (MAL) algorithms. Previous work has shown that UCB, M3, S or Exp3 algorithms have good behaviours on average in RMG. Besides, hedging algorithms have been shown to be effective on predi…
Meta-algorithm selection aims to choose the best algorithm selector for a given problem instance.
problem Selecting the best algorithm selector for a specific problem instance.
method Apply algorithm selection to the selection of other algorithms (meta-algorithm selection).
result Meta-algorithm selection can be beneficial in some cases but faces challenges in solving the meta-level problem.
Proposes CLRS benchmark to evaluate algorithmic reasoning.
problem Difficulty in transferring results across publications due to targeted algorithmic data.
method Develops a comprehensive benchmark covering various algorithmic tasks.
result Demonstrates performance of algorithmic reasoning baselines on the CLRS benchmark.
Combines multiple bandit algorithms to create a nearly optimal single algorithm.
problem Designing a single bandit algorithm that performs nearly as well as the best individual algorithm in a stochastic environment.
method Develops two general corralling algorithms that achieve favorable regret guarantees.
result The regret of the corralling algorithms is no worse than the best individual algorithm's performance.
We propose accelerated randomized coordinate descent algorithms for stochastic optimization and online learning. Our algorithms have significantly less per-iteration complexity than the known accelerated gradient algorithms. The proposed algorithms for online learning have better regret performance than the known rando…
The exchange algorithm is studied for its convergence and asymptotic variance.
problem Theoretical limitations of the exchange algorithm in sampling from doubly-intractable distributions.
method Theoretical analysis of the exchange algorithm's convergence speed and asymptotic variance.
result The exchange algorithm converges at a geometric rate and satisfies a Central Limit Theorem.
New algorithms optimize algorithm parameters in online settings with reduced computational costs.
problem Optimizing algorithm parameters in online settings with volatile and discontinuous losses.
method Developed semi-bandit optimization algorithms that leverage extra information to reduce computational costs.
result Achieved regret bounds as good as full-information feedback with significantly less computational effort.
Bayesian networks (BN) are used in a big range of applications but they have one issue concerning parameter learning. In real application, training data are always incomplete or some nodes are hidden. To deal with this problem many learning parameter algorithms are suggested foreground EM, Gibbs sampling and RBE algori…
No algorithm outperforms uniform sampling in A/B testing.
problem Identifying the best arm in A/B testing with fixed budget.
method Introducing consistent and stable algorithms, deriving lower bounds, and proving optimality of uniform sampling.
result No algorithm performs better than uniform sampling in A/B testing.
Improves algorithm selection for thousands of candidates using dyadic features.
problem Selecting the best algorithm from a large set of candidates for specific problems.
method Proposes extreme algorithm selection (XAS) with dyadic feature representation.
result Improves significantly over current state of the art in various metrics.
Combines online learning algorithms to achieve better performance.
problem Improving online learning algorithms with varying guarantees.
method Adding iterates of two parameter-free algorithms to create a new algorithm with improved regret.
result Generates efficient algorithms that adapt to multiple norms and maintain dimension-free guarantees.
New ELM algorithms reduce computation time and complexity.
problem Efficient computation of extreme learning machine (ELM) algorithms.
method Developed inverse-free ELM algorithms using recursive matrix inverse and inverse LDL' factorization.
result Proposed algorithms significantly reduce computational complexity.