Dynamic segmentation algorithm improves NMT performance by favoring character-level processing.
problem Suboptimal static segmentation choices in NMT systems.
method Adaptive Computation Time algorithm for dynamic segmentation, trainable end-to-end.
result The model prefers character-level processing when given the freedom to navigate different segmentation levels.
Deep transformer models outperform RNNs in character-level language modeling.
problem Improving character-level language modeling performance.
method A deep (64-layer) transformer model with fixed context and auxiliary losses.
result Achieved state-of-the-art performance on text8 and enwik8 benchmarks.
End-to-end ASR model combines word and character representation for improved performance.
problem Difficulty in training with word-level supervision due to sparsity of examples.
method Multi-task learning framework combining word and character representations.
result Improved word-error rate (WER) by interpolating between word-level and character-level models.
Charmer improves character-level adversarial attacks for language models.
problem Efficiency and effectiveness of character-level adversarial attacks for language models.
method Query-based adversarial attack method that maintains semantic similarity.
result Charmer achieves high attack success rate and similar adversarial examples.
This paper reports the performances of shallow word-level convolutional neural networks (CNN), our earlier work (2015), on the eight datasets with relatively large training data that were used for testing the very deep character-level CNN in Conneau et al. (2016). Our findings are as follows. The shallow word-level CNN…
The paper improves part-of-speech tagging with multi-task learning and character-level word representations.
problem Improving part-of-speech tagging accuracy.
method Developed a new character-level word representation using feedforward neural network, pretraining with existing word vectors, and an additional prediction of neighbour labels as an auxiliary loss.
result The methods significantly improved POS tagging performance on English and Russian languages.
Paper compares training and sampling schemes for character-level RNNs.
problem Optimizing training and sampling schemes for character-level RNNs.
method Tested various training and sampling schemes on different datasets and architectures.
result The choice of training and sampling scheme affects stability, performance, and implementation effort.
Deep neural nets predict click-through rates for sponsored search ads.
problem Predicting click-through rates for sponsored search ads.
method Two novel deep convolutional neural network approaches at character and word levels.
result Deep models significantly outperform baseline models and improve click-through rate prediction accuracy.
New models improve language generation by sharing intermediate states.
problem Language models struggle with past mistakes in sequence generation.
method Integrate second-order terms in hidden-state update, sharing intermediate states.
result Shared parametrization improves language modeling performance.
Synthetic noise training improves machine translation robustness to spelling mistakes.
problem Making machine translation robust to spelling mistakes and natural noise.
method Training on synthetic noise to improve robustness to natural noise.
result Training on synthetic noise improves robustness to natural noise without diminishing performance on clean text.
We introduce multiplicative LSTM (mLSTM), a recurrent neural network architecture for sequence modelling that combines the long short-term memory (LSTM) and multiplicative recurrent neural network architectures. mLSTM is characterised by its ability to have different recurrent transition functions for each possible inp…
Neural machine translation aims at building a single large neural network that can be trained to maximize translation performance. The encoder-decoder architecture with an attention mechanism achieves a translation performance comparable to the existing state-of-the-art phrase-based systems on the task of English-to-Fr…
End-to-end character-level model for text generation without delexicalization.
problem Generating text without delexicalization and tokenization.
method Character-level sequence-to-sequence model with attention mechanism, copy mechanism, and transfer learning.
result Competitive performance in text generation metrics.
Adaptive attention span extends Transformer's context size.
problem Limited context size in Transformers limits model performance.
method A self-attention mechanism that learns optimal attention span.
result State-of-the-art performance on character-level language modeling.
A transformer model improves spell correction with hierarchical attention.
problem Improving spell correction accuracy and speed.
method Multi encoder-single decoder transformer architecture with hierarchical attention.
result Significant improvement in CER, WER, and SER error rates.
We propose a generalization of neural network sequence models. Instead of predicting one symbol at a time, our multi-scale model makes predictions over multiple, potentially overlapping multi-symbol tokens. A variation of the byte-pair encoding (BPE) compression algorithm is used to learn the dictionary of tokens that …
A deep learning approach generates math word problems in multiple languages.
problem Template-based mechanisms for generating mathematical word problems lack customizability and creativity.
method Character Level Long Short Term Memory Network (LSTM) and POS tags are used to generate and resolve constraints in generated problems.
result The approach generates accurate math word problems in English and Sinhala with over 90% accuracy.
We stabilize the activations of Recurrent Neural Networks (RNNs) by penalizing the squared distance between successive hidden states' norms. This penalty term is an effective regularizer for RNNs including LSTMs and IRNNs, improving performance on character-level language modeling and phoneme recognition, and outperfor…
AvaGrad optimizes vision tasks by decoupling learning rate and adaptability.
problem Improving optimization methods for vision tasks.
method Derives AvaGrad, a new optimizer that decouples learning rate and adaptability.
result AvaGrad outperforms SGD on vision tasks when adaptability is properly tuned.
Classifies CAD model descriptions and names from product websites.
problem Distinguishing product descriptions from other text and identifying product names.
method Paragraph vectors, character-level LSTM, word embeddings LSTM tagger.
result Promising results for distinguishing product descriptions and names.
Hybrid approach combines ASTs and deep learning for PowerShell malware detection.
problem Detecting malicious PowerShell scripts effectively.
method Combining static program analysis (ASTs) with deep learning.
result Preliminary results show promising classification of PowerShell scripts by family type.
In this paper, we propose a probabilistic parsing model, which defines a proper conditional probability distribution over non-projective dependency trees for a given sentence, using neural representations as inputs. The neural network architecture is based on bi-directional LSTM-CNNs which benefits from both word- and …
Byte-level machine translation outperforms embedding-based methods.
problem Improving machine translation without embedding layers.
method Replacing embedding layers with one-hot representations of bytes, and using decoder-input dropout.
result Byte-to-byte machine translation achieves BLEU scores comparable to character-level and subword-level models.
We describe a simple neural language model that relies only on character-level inputs. Predictions are still made at the word-level. Our model employs a convolutional neural network (CNN) and a highway network over characters, whose output is given to a long short-term memory (LSTM) recurrent neural network language mo…
State-of-the-art sequence labeling systems traditionally require large amounts of task-specific knowledge in the form of hand-crafted features and data pre-processing. In this paper, we introduce a novel neutral network architecture that benefits from both word- and character-level representations automatically, by usi…
Recurrent Neural Networks (RNNs) have long been recognized for their potential to model complex time series. However, it remains to be determined what optimization techniques and recurrent architectures can be used to best realize this potential. The experiments presented take a deep look into Hessian free optimization…
The study examines how character and word-level representations improve sentence-level tasks.
problem Improving the quality of word and sentence representations using character-level information.
method Feature-wise sigmoid gating mechanism for combining character and word-level representations.
result Modeling characters improves final word and sentence representations, especially for less frequent words.
TensorFuzz automates neural network debugging with fuzzing.
problem Debugging neural networks for rare inputs is challenging.
method Coverage-guided fuzzing with approximate nearest neighbor algorithms.
result TensorFuzz discovers numerical errors and undesirable behavior.
A planning mechanism improves sequence-to-sequence model performance.
problem Improving sequence-to-sequence model performance for tasks like translation and graph analysis.
method Developed a model that plans ahead using attention and differentiable operations.
result Outperforms strong baselines on character-level translation, graph analysis, and question generation.
Trellis networks improve sequence modeling performance.
problem Sequence modeling challenges.
method Temporal convolutional network with weight tying and direct input injection.
result Trellis networks outperform state-of-the-art methods on benchmarks.
Discrete flows extend normalizing flows to discrete data, improving various applications.
problem Applying normalizing flows to discrete data distributions.
method Developed discrete autoregressive and bipartite flows, showing their effectiveness on various discrete data tasks.
result Discrete autoregressive flows outperform autoregressive baselines on synthetic discrete distributions and Potts models.
The Teacher Forcing algorithm trains recurrent networks by supplying observed sequence values as inputs during training and using the network's own one-step-ahead predictions to do multi-step sampling. We introduce the Professor Forcing algorithm, which uses adversarial domain adaptation to encourage the dynamics of th…
SmoothLLM defends LLMs from jailbreaking attacks by randomly perturbing inputs.
problem Adversaries can fool large language models into generating objectionable content.
method SmoothLLM randomly perturbs multiple copies of a prompt and aggregates predictions to detect adversarial inputs.
result SmoothLLM sets the state-of-the-art for robustness against various jailbreak attacks.
In this work, we propose a novel recurrent neural network (RNN) architecture. The proposed RNN, gated-feedback RNN (GF-RNN), extends the existing approach of stacking multiple recurrent layers by allowing and controlling signals flowing from upper recurrent layers to lower layers using a global gating unit for each pai…
Paper investigates Lipschitz constants of self-attention modules in neural networks.
problem Lipschitz constants of self-attention modules in neural networks.
method Proved standard dot-product self-attention is not Lipschitz for unbounded input domain. Proposed L2 self-attention that is Lipschitz. Derived upper bound on L2 self-attention's Lipschitz constant.
result Proved standard self-attention is not Lipschitz for unbounded input domain and proposed an alternative L2 self-attention that is Lipschitz.
A new algorithm reduces the computational cost of RTRL while maintaining performance.
problem Large computational costs in RTRL for learning long-term dependencies.
method Optimal Kronecker-Sum Approximation (OK) of RTRL.
result OK is optimal for a class of RTRL approximations and matches TBPTT in performance.
A new model uses normalizing flows for discrete sequences, improving generation speed.
problem Modeling discrete sequences like text using normalizing flows poses challenges.
method Proposes a VAE-based model with autoregressive and non-autoregressive flow architectures.
result Flow-based models can match or improve on autoregressive baselines for discrete sequence tasks.
The paper uses attention networks for character-based handwritten text transcription.
problem Handwritten text recognition with improved character-level alignment.
method Attentional encoder-decoder networks trained on character sequences, comparing different activation functions.
result Softmax attention provides more precise character alignment than sigmoid attention.
Bayesian approach optimizes in-context learning for state space models.
problem Optimizing in-context learning for state space models.
method Bayesian optimal sequential prediction over latent sequence tasks.
result Bayesian optimal predictor converges to posterior predictive mean.
GTI network learns linguistic features for multi-task sequence tagging.
problem Improving neural model performance on multi-task sequence tagging without explicit features.
method GTI network with neural gate modules to learn relations between tasks.
result GTI network outperforms baselines on chunking and NER tasks.
HW2MP-GAN tackles ancient handwritten text recognition.
problem Automatic text recognition from ancient handwritten records.
method Conditional Generative Adversarial Network (HW2MP-GAN) with Sliced Wasserstein distance and U-Net architectures.
result HW2MP-GAN outperforms state-of-the-art models in image-to-image translation and handwritten recognition.
DeepOBS benchmarks deep learning optimizers.
problem Quantitative evaluation of deep learning optimizers.
method Python package with realistic benchmarks and back-ends.
result Automates benchmarking of stochastic optimizers.
Single neuron with ADA learns XOR and outperforms other functions.
problem Classifying linearly non-separable data.
method Proposed a new artificial neuron with apical dendrite activation.
result ADA function achieves 100% accuracy on XOR and superior performance on benchmark datasets.
Monte Carlo method trains deep neural networks without gradients.
problem Vanishing and exploding gradients in backpropagation.
method Randomly mutate parameters, keep if loss decreases.
result Gradient-free method trains deep networks effectively.
Telephonetic improves robustness of neural language models to ASR errors.
problem Handling ASR errors and semantic noise in neural language models trained on written text.
method Data augmentation framework using character-level and embedding space perturbations.
result Achieves state-of-the-art perplexity of 37.49 on Penn Treebank corpus.
Cloned HMMs efficiently learn variable order sequences without local minima issues.
problem Learning long-term temporal structure in sequences.
method Constrained HMMs with a sparsity structure that maps hidden states deterministically to emissions.
result Cloned HMMs can model temporal dependencies at arbitrarily long distances and recognize contexts with 'holes'.
RUM improves RNN's long-term memory by using unitary matrices.
problem Limited capacity of RNN to manipulate long-term memory.
method Proposes Rotational Unit of Memory (RUM) with unitary matrices.
result RUM learns long-term dependencies and improves state-of-the-art results.
RNN models classify poem meters from plain text with high accuracy.
problem Classifying poem meters from plain text.
method Character-level encoding, RNN models, no feature handcrafting.
result 96.38% accuracy for Arabic poems, 82.31% for English poems.