Benchmark evaluates LLM trading agents by masking identifiers to prevent memory leaks.
problem Evaluate LLM trading agents without relying on market memory or noise.
method Data-side masking protocol, Barra-style performance attribution framework.
result LLM agents' returns are largely explained by market and style exposure, not stock selection.
Transformers predict price movements from limit order books.
problem Predicting price movements from limit order books.
method Causal convolutional network with masked self-attention.
result Significantly outperforms existing architectures on FI-2010 dataset.
This research uncovers high-performing subnetworks in deep GNNs without training.
problem Challenges in applying SLTH to deeper GNNs with high memory requirements.
method Introduces Multicoated Supermasks (M-Sup) and Multi-Stage Folding for GNNs.
result Uncovered untrained recurrent networks with performance similar to trained models.
The memory capacity of linear echo state networks is accurately calculated using new numerical methods.
problem Numerical evaluations of memory capacity in recurrent neural networks often contradict theoretical bounds.
method Developed robust numerical approaches exploiting MC neutrality with respect to the input mask matrix.
result Memory curves fully agree with theory when using the proposed methods.
Global Memory Augmentation (GMAT) improves Transformer performance on long documents.
problem Large memory requirements of Transformer pairwise dot-product attention for long sequences.
method Integrates a dense global memory of length M into sparse Transformer blocks.
result Significant improvement on various tasks, including synthetic tasks, masked language modeling, and reading comprehension.
Stochastic attention learns to retrieve and generate from memory without training.
problem Learning to retrieve and generate from memory efficiently.
method Langevin dynamics on modern Hopfield energy for stochastic attention.
result Stochastic attention can retrieve and generate from memory without training, matching gold standards.
Efficient sparse modern Hopfield models are introduced for memory retrieval and learning tasks.
problem Efficient modern Hopfield models for memory retrieval and learning tasks.
method Nonparametric interpretation of Hopfield models as a regression problem.
result Sparse-structured modern Hopfield models with sub-quadratic complexity.
This paper explores four different visualization techniques for long short-term memory (LSTM) networks applied to continuous-valued time series. On the datasets analysed, we find that the best visualization technique is to learn an input deletion mask that optimally reduces the true class score. With a specific focus o…
Non-autoregressive transformer improves speech recognition speed and accuracy.
problem Reducing inference computation cost in speech recognition.
method Two non-autoregressive transformer structures (A-CMLM and A-FMLM) trained with masked tokens and iterative prediction during inference.
result Non-autoregressive transformer can match autoregressive transformer performance with 7x speedup.
PMI-Masking improves MLM pretraining by masking correlated spans efficiently.
problem Uniform token masking leads to inefficient and suboptimal performance in MLMs.
method PMI-Masking uses Pointwise Mutual Information to mask n-grams with high collocation.
result PMI-Masking reaches half the training time and improves performance.
Neural networks with low-precision weights and activations offer compelling efficiency advantages over their full-precision equivalents. The two most frequently discussed benefits of quantization are reduced memory consumption, and a faster forward pass when implemented with efficient bitwise operations. We propose a t…
We propose to execute deep neural networks (DNNs) with dynamic and sparse graph (DSG) structure for compressive memory and accelerative execution during both training and inference. The great success of DNNs motivates the pursuing of lightweight models for the deployment onto embedded devices. However, most of the prev…
Convolutional sparse representations are a form of sparse representation with a structured, translation invariant dictionary. Most convolutional dictionary learning algorithms to date operate in batch mode, requiring simultaneous access to all training images during the learning process, which results in very high memo…
ST-MTM models complex time series by decomposing and masking seasonal and trend components.
problem Forecasting complex time series with intricate temporal variations.
method Seasonal-Trend Decomposition with Masking and Contrastive Learning.
result ST-MTM achieves superior forecasting performance compared to existing methods.
Expands MLM by masking token positions, improving performance and convergence.
problem Improving language model performance and convergence.
method Masking token positions along with [MASK] tokens, using a fully connected classifier stage.
result Shows .3% improvement and 50% faster convergence for BERT Base with position masking.
SMART training improves mask-predict translations.
problem Closing the performance gap between semi-autoregressive and autoregressive models.
method SMART training method for conditional masked language models.
result SMART-trained models produce higher-quality translations.
New method reduces diffusion model function evaluations for discrete data.
problem High computational burden in generating samples from masked diffusion models.
method Modified causal attention mask and speculative sampling mechanism for non-factorized predictions.
result Achieved ~2x reduction in required network forward passes.
Proposes a proportional masking strategy for better tabular data imputation.
problem Heterogeneity of tabular data disrupts uniform random masking in MAEs.
method Computes missingness statistics, generates proportional masks, uses MLP token mixing.
result Proportional masking preserves missingness distribution, improves imputation performance.
New analysis reveals masked self-supervised learning's effectiveness in extracting data structure.
problem Analyzing masked self-supervised learning in high-dimensional data.
method Developed precise high-dimensional analysis of masked modeling objectives.
result Identified phase transitions and structured regimes for masked self-supervised learning.
Pruning is an efficient model compression technique to remove redundancy in the connectivity of deep neural networks (DNNs). Computations using sparse matrices obtained by pruning parameters, however, exhibit vastly different parallelism depending on the index representation scheme. As a result, fine-grained pruning ha…
DiffusionBlocks trains neural networks by breaking them into independent blocks, reducing memory usage.
problem Memory bottlenecks in end-to-end neural network training.
method Transforming transformer-based networks into independent trainable blocks via a denoising process.
result Independent block-wise training matches end-to-end training performance while reducing memory requirements.
This paper connects masked pre-training to Bayesian model selection.
problem Understanding the success of masked pre-training and its generalization.
method The paper shows masked pre-training corresponds to maximizing the marginal likelihood.
result Masked pre-training with a suitable scoring function maximizes the marginal likelihood.
Optimizes pruning masks for neural networks using probabilistic fine-tuning and PAC-Bayes bounds.
problem Improving neural network performance through adaptive pruning of weights.
method Optimizes stochastic pruning masks by minimizing expected loss, considering data-adaptive regularization and feature alignment.
result Probabilistic fine-tuning leads to improved test error over baseline methods in neural networks.
We consider the problem of selecting an optimal mask for an image manifold, i.e., choosing a subset of the pixels of the image that preserves the manifold's geometric structure present in the original data. Such masking implements a form of compressive sensing through emerging imaging sensor platforms for which the pow…
Recent object detectors use four-coordinate bounding box (bbox) regression to predict object locations. Providing additional information indicating the object positions and coordinates will improve detection performance. Thus, we propose two types of masks: a bbox mask and a bounding shape (bshape) mask, to represent t…
Paper proposes a machine learning framework for VLSI mask optimization.
problem Costly VLSI mask optimization due to complex processes.
method Heterogeneous OPC framework using machine learning.
result Demonstrates efficiency and effectiveness of the proposed framework.
The paper investigates the effects of invalid action masking in policy gradient algorithms.
problem Invalid actions in policy gradient algorithms can lead to suboptimal performance.
method The paper provides theoretical justification and empirical demonstrations of the importance of invalid action masking.
result Invalid action masking is crucial as the number of invalid actions increases.
WaveCRN improves E2E speech enhancement with efficient CNN and SRU.
problem Efficiently modeling speech locality and sequential properties for E2E speech enhancement.
method WaveCRN uses a CNN for speech locality and SRU for temporal sequential modeling, with RFM for noise suppression.
result WaveCRN outperforms state-of-the-art approaches with reduced complexity and inference time.
Develops masks to explain neural network predictions.
problem Improving neural network interpretability for various applications.
method Creates explanation masks for pre-trained networks using a secondary network.
result Demonstrates the effectiveness of the method across different types of networks.
Improved YOLOv5 model detects mask-wearing with enhanced accuracy.
problem Detecting mask-wearing in high traffic public places.
method Improved YOLOv5l with Multi-Head Attentional Self-Convolution, Swin Transformer Block, I-CBAM module, and enhanced feature fusion.
result 1.1% improvement in mAP(0.5) and 1.3% improvement in mAP(0.5:0.95) compared to YOLOv5l.
Simplified masked diffusion models improve discrete data generation.
problem Complex model formulations and unclear relationships hinder discrete data generative modeling.
method Developed a simple and general framework for masked diffusion models.
result Models trained on OpenWebText surpass prior diffusion language models and outperform autoregressive models.
Language models trained on chess board states outperform those on moves, even with causal masking.
problem Applying causal masking to spatial data for training unimodal language models.
method Trained bidirectional and causal self-attention models on both spatial (board-based) and sequential (move-based) chess data.
result Models trained on spatial board states achieve stronger playing strength than those trained on sequential data, even with causal masking.
Masking diffusion outperforms other discrete diffusion models by incorporating jump times into the model.
problem Improving the performance of discrete diffusion models.
method Conditioning on the jump schedule of discrete Markov processes.
result Schedule-conditioned discrete diffusion (SCUD) models outperform classical and masking diffusion models.
Secure aggregation for buffered asynchronous federated learning without TEEs.
problem Privacy and convergence in buffered asynchronous federated learning.
method Developed a new protocol (BASecAgg) that ensures privacy without TEEs by carefully designing masks.
result BASecAgg achieves similar convergence guarantees as FedBuff without TEEs.
Study different masking schemes for a universal marginaliser.
problem Understand how well a neural approximator learns conditional distributions.
method Compare networks trained with various masking schemes.
result Neural approximators perform differently based on the masking scheme.
We develop mask iterative hard thresholding algorithms (mask IHT and mask DORE) for sparse image reconstruction of objects with known contour. The measurements follow a noisy underdetermined linear model common in the compressive sampling literature. Assuming that the contour of the object that we wish to reconstruct i…
The ConditionaL Neural Networks (CLNN) and the Masked ConditionaL Neural Networks (MCLNN) exploit the nature of multi-dimensional temporal signals. The CLNN captures the conditional temporal influence between the frames in a window and the mask in the MCLNN enforces a systematic sparseness that follows a filterbank-lik…
Unified framework for ICL in causal and masked models.
problem Understanding ICL in masked language models and comparing it to causal models.
method Developed a statistical learning framework representing context by empirical measure and predicting using context and query.
result Upper bounds for masked and autoregressive objectives under Wasserstein-type regularity conditions.
Most machine translation systems generate text autoregressively from left to right. We, instead, use a masked language modeling objective to train a model to predict any subset of the target words, conditioned on both the input text and a partially masked target translation. This approach allows for efficient iterative…
Novel bio-inspired masking for robust speech emotion recognition.
problem Noise degradation in speech emotion recognition.
method Cochlear cepstrogram-based contrastive learning with temporal and frequency masking.
result Improved speech emotion recognition performance on K-EmoCon benchmark.
This paper analyzes MaskGIT sampler and introduces a moment sampler for faster masked diffusion sampling.
problem Efficiently sampling from masked diffusion models.
method Theoretical analysis of MaskGIT sampler, introduction of moment sampler, and two innovations for improving choose-then-sample efficiency.
result The moment sampler is an asymptotically equivalent, more interpretable alternative to MaskGIT.
Modulating masks improve lifelong reinforcement learning.
problem Catastrophic forgetting and task interference in lifelong reinforcement learning.
method Adapted modulating masks for deep lifelong reinforcement learning (LRL) with PPO and IMPALA agents.
result Superior performance in both discrete and continuous RL tasks compared to LRL baselines.
PCNN prunes CNN weights efficiently for hardware acceleration.
problem Efficiently compressing CNN models for hardware acceleration.
method PCNN uses a novel Sparsity Pattern Mask (SPM) to encode and prune weights.
result PCNN achieves up to 8.4X compression with minimal accuracy loss.
Enhanced Sampling Scheme improves masked generative modeling.
problem Limitations of existing sampling schemes in masked non-autoregressive generative modeling.
method ESS consists of three stages: Naive Iterative Decoding, Critical Reverse Sampling, and Critical Resampling.
result ESS achieves significant performance gains in unconditional and class-conditional sampling.
Structured State-Space Duality connects SSMs to masked attention.
problem Connecting SSMs and attention mechanisms for efficient modeling.
method Formalizing and generalizing SSD from scalar-identity to diagonal state matrices.
result Diagonal SSMs match training complexity lower bounds and support richer dynamics.
Study improves healthcare time series imputation by considering structured missingness.
problem Structured missingness in clinical data impacts time series imputation models.
method Analysis of different masking strategies on imputation methods using PhysioNet Challenge 2012 dataset.
result Masking choices significantly affect imputation accuracy and clinical prediction.
Training-free method improves text-to-image generation quality.
problem Error accumulation in simultaneous token updates of masked diffusion models.
method Training-free self-correction framework exploiting inductive biases.
result Significantly improved generation quality on text-to-image tasks.
Deep learning based speech enhancement and source separation systems have recently reached unprecedented levels of quality, to the point that performance is reaching a new ceiling. Most systems rely on estimating the magnitude of a target source by estimating a real-valued mask to be applied to a time-frequency represe…