New method learns output embeddings for structured prediction.
problem Structured prediction with output embeddings.
method Jointly learns output embedding and regression function.
result Structured predictor is a consistent estimator with smaller complexity.
Method learns audio embeddings with contextualized tags.
problem Align audio and tags for cross-modal tasks.
method Audio autoencoder, word embeddings, multi-head self-attention, contrastive loss.
result Multi-head self-attention improves audio representations.
Proposes a deep tree-ensemble model for multi-output prediction.
problem Lack of efficient solutions for multi-output prediction.
method Integrates tree-embeddings into deep tree-ensembles for structured output prediction.
result Superior performance in multi-label classification and multi-target regression tasks.
We present the Procrustes measure, a novel measure based on Procrustes rotation that enables quantitative comparison of the output of manifold-based embedding algorithms (such as LLE (Roweis and Saul, 2000) and Isomap (Tenenbaum et al, 2000)). The measure also serves as a natural tool when choosing dimension-reduction …
New learning rates for embeddings in RKHSs, even when the target is not Hilbert-Schmidt.
problem Applying conditional mean embeddings to complex ML/RL settings with infinite-dimensional RKHSs.
method Developed novel learning rates using interpolation theory for RKHSs, derived explicit adaptive rates for sample estimator.
result Achieved uniform convergence rates in the output RKHS for certain parameter regimes.
Wide Boosting improves GB's performance on multivariate output tasks.
problem Lack of flexibility in fitting probabilistic multi-dimensional outputs.
method Inserts matrix multiplication between GB output and loss function.
result Wide Boosting outperforms Gradient Boosting on multivariate output tasks.
RESTA defends LLMs against jailbreaking attacks by adding random noise to embeddings.
problem Vulnerability of LLMs to jailbreaking attacks that generate harmful outputs.
method Adds random noise to embedding vectors and aggregates during token generation.
result RESTA achieves superior robustness versus utility tradeoffs compared to baseline defenses.
New recursive algorithm estimates conditional kernel mean embeddings in Hilbert space.
problem Estimating conditional distributions in RKHS for supervised learning.
method Recursive algorithm in L2 space for conditional kernel mean map. result Strong L2 consistency of recursive estimator proved. MCSAE improves speaker embedding by focusing on both high- and low-level features.
problem Reduced effect of low-level features in speaker embedding encoding.
method Masked cross self-attentive encoding using ResNet with multi-layer aggregation and random masking regularization.
result Improved speaker embedding with equal error rate of 2.63% and minimum detection cost function of 0.1453.
Paper studies t-SNE convergence with generalized kernels.
problem Understanding convergence of t-SNE with generalized kernels.
method Concrete formulation of generalized kernels, proving convergence to an equilibrium distribution.
result t-SNE converges to an equilibrium distribution under certain conditions for generalized kernels.
We examine the algebraic and geometric properties of a uni-directional GRU and word embeddings trained end-to-end on a text classification task. A hyperparameter search over word embedding dimension, GRU hidden dimension, and a linear combination of the GRU outputs is performed. We conclude that words naturally embed t…
Gradient-trained embeddings select important tokens in text.
problem Understanding the theoretical structure of token embeddings.
method One-layer softmax attention model with gradient descent.
result Gradient-trained embeddings align with important tokens based on frequency.
Complex textual information extraction tasks are often posed as sequence labeling or \emph{shallow parsing}, where fields are extracted using local labels made consistent through probabilistic inference in a graphical model with constrained transitions. Recently, it has become common to locally parametrize these models…
Deep neural networks are composed of layers of parametrised linear operations intertwined with non linear activations. In basic models, such as the multi-layer perceptron, a linear layer operates on a simple input vector embedding of the instance being processed, and produces an output vector embedding by straight mult…
Earth observation embeddings can convert discrete biome maps into continuous representations that better capture ecological variation.
problem Biome maps impose categorical boundaries that compress continuous variation in biotic communities.
method Fit a linear classifier on Earth observation embeddings to predict biome labels.
result Continuous biome representation outperforms discrete biome labels for predicting species occurrence.
We show how to learn low-dimensional representations (embeddings) of patient visits from the corresponding electronic health record (EHR) where International Classification of Diseases (ICD) diagnosis codes are removed. We expect that these embeddings will be useful for the construction of predictive statistical models…
Proposes cone embedding for better graph hierarchical structure representation.
problem Lack of natural and interpretable hierarchical indicators in graph embeddings.
method Metric cone embedding method to capture hierarchical structure.
result Extracts hierarchical structure from other graph embedding outputs.
End-to-end dialogue model learns from joint embeddings and user intent.
problem Challenges in reasoning and incorporating state-full knowledge in goal-oriented dialogues.
method Proposes an RNN-based end-to-end encoder-decoder architecture trained with joint embeddings and multi-task learning.
result Improves task-oriented dialogue system performance as shown by BLEU score evaluation.
Paper proposes a novel graph recovery attack from node embeddings.
problem Privacy risks of integrating graph embeddings with machine learning pipelines.
method Model-agnostic graph recovery attack exploiting preserved structural information in node embeddings.
result Adversaries can recover graph edges with decent accuracy from node embeddings alone.
Robustly computes intrinsic coordinates on point clouds using resampling and averaging.
problem Computing intrinsic coordinates on noisy or outlier-prone point clouds.
method Subsample data, vary hyperparameters, cluster candidate embeddings, identify representative embeddings, and average them using Procrustes analysis.
result Robust to noise and outliers, validated on synthetic and real data.
This paper proposes a new dimensionality reduction algorithm named branching embedding (BE). It converts a dendrogram to a two-dimensional scatter plot, and visualizes the inherent structures of the original high-dimensional data. Since the conversion part is not computationally demanding, the BE algorithm would be ben…
Topic models are frequently used in machine learning owing to their high interpretability and modular structure. However, extending a topic model to include a supervisory signal, to incorporate pre-trained word embedding vectors and to include a nonlinear output function is not an easy task because one has to resort to…
Despite great popularity of applying softmax to map the non-normalised outputs of a neural network to a probability distribution over predicting classes, this normalised exponential transformation still seems to be artificial. A theoretic framework that incorporates softmax as an intrinsic component is still lacking. I…
This work analyzes label embedding for large multiclass classification problems.
problem Label embedding for large multiclass classification problems.
method Analysis of label embedding in extreme multiclass classification, presenting an excess risk bound and showing a trade-off between computational and statistical efficiency.
result The statistical penalty for label embedding vanishes with sufficiently low coherence under the Massart noise condition.
Generative Adversarial Nets (GANs) and Variational Auto-Encoders (VAEs) provide impressive image generations from Gaussian white noise, but the underlying mathematics are not well understood. We compute deep convolutional network generators by inverting a fixed embedding operator. Therefore, they do not require to be o…
NLE embeds labels for domain adaptation with neural networks.
problem Adapting deep neural networks with unpaired source and target domain data.
method Distill source-domain knowledge into l-vectors, soft targets for adaptation.
result 14.1% relative word error rate reduction over direct re-training.
Machine learning algorithms are optimized to model statistical properties of the training data. If the input data reflects stereotypes and biases of the broader society, then the output of the learning algorithm also captures these stereotypes. In this paper, we initiate the study of gender stereotypes in {\em word emb…
The paper uncovers symmetries in large language models through layer-peeled optimization.
problem Understanding geometric structure in large language model weights and context embeddings.
method Constrained layer-peeled optimization program to analyze symmetries in next-token distributions.
result Symmetries in target next-token distributions are transferred to optimal model weights and context embeddings.
E2M predicts metric space outputs using deep learning.
problem Predicting non-Euclidean outputs like distributions and matrices.
method Weighted Fréchet means over learned weights.
result E2M achieves state-of-the-art performance across various outputs.
Optimizes t-SNE for high-dimensional data with random projections.
problem High computational cost of t-SNE for high-dimensional data.
method Use random projections to reduce high-dimensional data to a few dimensions, then apply t-SNE.
result Random projections preserve clustering while significantly reducing t-SNE runtime.
The study shows removing fully connected output layers improves efficiency without sacrificing performance.
problem Large number of parameters in fully connected layers for high-category datasets.
method Examined architectures replacing fully connected output layers with fixed layers and compared performance.
result Fixed classifiers offer no additional benefit over removing the output layer and its parameters.
A new framework for structured prediction on non-vectorial spaces.
problem Structured prediction on non-vectorial output spaces.
method Defining a suitable geometry for implicit loss functions.
result Efficient algorithmic framework with sharp statistical analysis.
In this paper, we propose a new pooling method called spatial pyramid encoding (SPE) to generate speaker embeddings for text-independent speaker verification. We first partition the output feature maps from a deep residual network (ResNet) into increasingly fine sub-regions and extract speaker embeddings from each sub-…
Improved sample efficiency with normalized RBF kernels in neural networks.
problem Learning more with less data in deep learning models.
method Two-phase method to train neural networks with normalized RBF kernels as output layer.
result Normalized RBF kernel networks achieve higher sample efficiency, compactness, and separability.
DeepWalk embeddings converge on SBM graphs, recovering cluster structure.
problem Theoretical guarantees for DeepWalk embeddings on complex graphs.
method Solving a nonconvex optimization problem using random walks.
result DeepWalk embeddings on SBM graphs recover cluster structure with high probability.
Paper tackles novelty detection in text classification.
problem Traditional text classification assumes known classes in testing, but often encounters unexpected instances.
method Converts problem to pair-wise matching, uses CNN with embedding matrices.
result Proposed method outperforms state-of-the-art baselines.
Study shows gMPNNs struggle with OOD link prediction in larger test graphs.
problem Inductive out-of-distribution link prediction in larger test graphs.
method Theoretical analysis and development of a gMPNN with structural pairwise embeddings.
result Structural node embeddings from gMPNNs converge to random guessing as test graphs grow.
Embedding principle explains loss landscape of deep neural networks.
problem Understanding the structure of loss landscapes in deep neural networks.
method Proposed an embedding principle that critical points of narrower DNNs can be embedded to critical points of wider DNNs.
result Wide DNNs are often attracted by highly-degenerate critical points embedded from narrower DNNs.
In the field of Natural Language Processing (NLP), we revisit the well-known word embedding algorithm word2vec. Word embeddings identify words by vectors such that the words' distributional similarity is captured. Unexpectedly, besides semantic similarity even relational similarity has been shown to be captured in word…
A new method reduces the complexity of tensor products from cubic to quadratic, improving both speed and accuracy.
problem Efficiently computing high-dimensional tensor products for polynomial kernels.
method Complex-to-Real (CtR) modification of sketches using complex random projections.
result Achieves state-of-the-art performance in accuracy and speed.
Embeddings in machine learning are low-dimensional representations of complex input patterns, with the property that simple geometric operations like Euclidean distances and dot products can be used for classification and comparison tasks. The proposed meta-embeddings are special embeddings that live in more general in…
Flexible embedding framework for diverse data types.
problem Limited applicability of existing embedding learning methods.
method A flexible framework using entity-relation-matrices and sampling mechanism.
result Framework outperforms state-of-the-art approaches in various tasks.
E-Stitchup augments pre-trained embeddings for better model performance and confidence calibration.
problem Improving classification accuracy and confidence calibration of models using pre-trained embeddings.
method Data augmentation methods inspired by Mixup combined with label softening.
result Significantly increased classification accuracy and reduced training time.
We propose several ways of reusing subword embeddings and other weights in subword-aware neural language models. The proposed techniques do not benefit a competitive character-aware model, but some of them improve the performance of syllable- and morpheme-aware models while showing significant reductions in model sizes…
MPVAE learns latent embeddings and label correlations for multi-label classification.
problem Challenging task of predicting multiple targets with label correlations.
method Proposes MPVAE, a novel framework that learns latent embedding spaces and label correlations using a Multivariate Probit model.
result MPVAE outperforms state-of-the-art methods on various application domains and is robust under noisy settings.
FreSh shifts model's initial frequency spectrum to match target signal, improving neural representation performance.
problem MLPs' low-frequency bias limits capturing high-frequency details accurately.
method FreSh selects embedding hyperparameters to align model's initial output spectrum with target signal's spectrum.
result FreSh improves performance across various neural representation methods and tasks with minimal computational overhead.
This paper presents an end-to-end text-independent speaker verification framework by jointly considering the speaker embedding (SE) network and automatic speech recognition (ASR) network. The SE network learns to output an embedding vector which distinguishes the speaker characteristics of the input utterance, while th…
In the last two decades, automatic extractive text summarization on lectures has demonstrated to be a useful tool for collecting key phrases and sentences that best represent the content. However, many current approaches utilize dated approaches, producing sub-par outputs or requiring several hours of manual tuning to …