Paper proposes graph-based separable transforms for video coding.
problem Improving video coding efficiency by better capturing residual block statistics.
method Derives graph-based separable transforms (GBSTs) from line graphs with weights determined by parameters.
result GBSTs achieve about 0.4% average coding gain over existing transforms in VVC.
New WildMix dataset and Spectro-Temporal Transformer model for better monoaural audio source separation.
problem Challenging monoaural audio source separation.
method Introducing WildMix dataset and Spectro-Temporal Transformer model (STT) with Spectro-Temporal Encoder (STE).
result STT outperforms previous baselines on the WildMix dataset.
Transformers can learn noisy linear systems with depth and IID data.
problem Learning noisy linear dynamical systems with transformers.
method Theoretical analysis of multi-layer and single-layer transformers with respect to L2-testing loss. result Single-layer transformers have a non-diminishing lower bound on approximation error, suggesting depth separation.
Neural networks use their hidden layers to transform input data into linearly separable data clusters, with a linear or a perceptron type output layer making the final projection on the line perpendicular to the discriminating hyperplane. For complex data with multimodal distributions this transformation is difficult t…
The paper extends optimal transport for linear separability of sheared distributions in supervised learning.
problem Learning on the space of probability measures using shifts and scalings.
method Embedding probability measures into L2 spaces using optimal transport, then applying regular machine learning techniques. result Sheared distributions can be linearly separated under certain conditions, with bounds on transformations.
Transforms data into separable subspaces for clustering.
problem Data is not always separable into subspaces.
method Embeds subspace clustering techniques into transform learning.
result Improves upon state-of-the-art clustering techniques.
Proposes a two-step method for sound source separation.
problem Improving sound source separation performance.
method First, learn a latent space transform. Second, train a separation module in the latent space.
result The proposed method achieves better performance than joint learning approaches.
Sensory data are often comprised of independent content and transformation factors. For example, face images may have shapes as content and poses as transformation. To infer separately these factors from given data, various ``disentangling'' models have been proposed. However, many of these are supervised or semi-super…
Constructs orthogonal coordinates in curved spaces.
problem Separating variables in curved spaces.
method Explicit construction of orthogonal coordinates and transformations.
result Explicit formulas for Killing tensors and Stäckel matrices.
Sharp conditions link separators to R-trees for space transformations.
problem Conditions for separators to form R-trees under group actions.
method Sharp conditions linking separators to R-trees for space transformations.
result Sharp conditions linking separators to R-trees for space transformations.
Shallow nonlinear networks can separate classes linearly with polynomially scaling width.
problem Understanding the linear separability of deep networks' features.
method Modeling inputs as a union of low-dimensional subspaces and using random weights and quadratic activations.
result Shallow nonlinear networks can achieve linear separation with polynomially scaling width.
Disentangled representation learning finds compact, independent and easy-to-interpret factors of the data. Learning such has been shown to require an inductive bias, which we explicitly encode in a generative model of images. Specifically, we propose a model with two latent spaces: one that represents spatial transform…
Recent deep learning approaches have achieved impressive performance on speech enhancement and separation tasks. However, these approaches have not been investigated for separating mixtures of arbitrary sounds of different types, a task we refer to as universal sound separation, and it is unknown how performance on spe…
In this paper, we presented a novel semi-supervised one-class classification algorithm which assumes that class is linearly separable from other elements. We proved theoretically that class is linearly separable if and only if it is maximal by probability within the sets with the same mean. Furthermore, we presented an…
CascadeXML improves multi-resolution learning for XMC with transformer features.
problem Learning subset labels from millions of choices with trade-offs between performance and computation.
method End-to-end multi-resolution learning pipeline using transformer multi-layer architecture.
result Significantly outperforms existing approaches on benchmark datasets.
We present a novel blind source separation (BSS) method, called information geometric blind source separation (IGBSS). Our formulation is based on the log-linear model equipped with a hierarchically structured sample space, which has theoretical guarantees to uniquely recover a set of source signals by minimizing the K…
This work introduces a transformation-based learner model for classification forests. The weak learner at each split node plays a crucial role in a classification tree. We propose to optimize the splitting objective by learning a linear transformation on subspaces using nuclear norm as the optimization criteria. The le…
Memory capacity of DAM scales exponentially with feature separation, unaffected by correlations.
problem Understanding how feature correlations impact DAM's capacity.
method Developed an empirical framework to analyze DAM's capacity under varying feature correlations and pattern separations.
result Memory capacity scales exponentially with feature separation, unaffected by correlations.
This paper compares Transformers and RNNs in various tasks, showing size differences.
problem Comparing representational capabilities of Transformers and RNNs across tasks.
method Analysis of differences in tasks like index lookup, nearest neighbor, and string equality.
result Size differences in Transformers and RNNs for various tasks.
SAVeD detects dataset versions without metadata, improving accuracy and separation.
problem Difficulty in identifying similar versions of structured datasets.
method Contrastive learning with modified SimCLR pipeline, generating and contrasting augmented table views.
result SAVeD achieves higher accuracy and separation scores on unseen tables.
IGSD separates task-specific content channels in transformer components by comparing activation replacement with zero ablation.
problem Mechanistic interpretability of transformer components
method IGSD: paired-intervention framework for comparing activation replacement with zero ablation
result IGSD identifies an early-layer content channel in transformer components that standard importance methods underestimate.
The paper proves local laws for non-separable sample covariance matrices.
problem Analyzing non-separable sample covariance matrices with dependent or nonlinearly transformed data.
method Tensor network framework for analyzing fluctuation averaging in the presence of higher-order cumulant structure.
result Optimal averaged local law and full anisotropic local law for non-separable sample covariance matrices.
Characterizes Bayesian networks up to unconditional equivalence.
problem Characterizing Bayesian networks up to unconditional equivalence.
method Transformational characterization via undirected graphs and specified moves.
result Two DAGs are in the same UEC if and only if one can be transformed into the other via a finite sequence of moves.
This paper explains a mechanism called phase collapse that improves image classification accuracy.
problem Understanding the role of non-linearities and convolutional filters in image classification.
method Demonstrates phase collapse as a mechanism that eliminates spatial variability and linearly separates classes.
result Phase collapse improves classification accuracy, while thresholding operators degrade performance.
Transformers excel at sparse token selection, surpassing FCNs in both worst and average cases.
problem Sparse token selection task
method One-layer transformer trained with gradient descent
result Transformers learn sparse token selection and exhibit strong out-of-distribution length generalization
GMC benchmark isolates retrieval in Transformers, revealing max-margin alignment.
problem Understanding how Transformers develop match-and-copy behavior on natural data.
method Introducing Gaussian Match-and-Copy (GMC) as a minimalist benchmark.
result Gradient descent drives parameters to diverge while aligning with max-margin separator.
We present two range characterizations for the attenuated geodesic X-ray transform defined on pairs of functions and one-forms on simple surfaces. Such characterizations are based on first isolating the range over sums of functions and one-forms, then separating each sub-range in two ways, first by implicit conditions,…
Improved U-Nets with various intermediate blocks enhance singing voice separation.
problem Improving singing voice separation accuracy using U-Net architectures.
method Implemented and compared U-Nets with different intermediate spectrogram transformation blocks.
result A specific block type achieves state-of-the-art SDR by 0.9 dB.
SAHMM-VAE separates sources adaptively using hidden Markov priors.
problem Unsupervised blind source separation.
method Source-wise adaptive Hidden Markov prior variational autoencoder.
result Different latent dimensions align with different source-specific temporal organizations.
New kernels allow learning from non-separable data.
problem Learning from non-separable data.
method Introducing entangled kernels and a two-step algorithm.
result Efficient algorithm for learning entangled kernels.
This study analyzes how one-layer transformers learn regular language recognition tasks.
problem Understanding how one-layer transformers solve regular language recognition tasks like even pairs and parity check.
method Theoretical analysis of training dynamics and gradient descent for a one-layer transformer.
result A one-layer transformer can solve even pairs directly but needs CoT for parity check. Training phases show rapid growth in attention layer followed by logarithmic growth in linear layer.
Enhanced Hopfield model boosts memory retrieval capacity.
problem Memory retrieval in modern Hopfield models with limited capacity.
method Introduces a learnable feature map transforming energy function into kernel space, minimizing separation loss for uniform memory distribution.
result Significant reduction in metastable states, enhancing memory capacity and retrieval accuracy.
In many state-of-the-art compression systems, signal transformation is an integral part of the encoding and decoding process, where transforms provide compact representations for the signals of interest. This paper introduces a class of transforms called graph-based transforms (GBTs) for video compression, and proposes…
Numerous algorithms are used for nonnegative matrix factorization under the assumption that the matrix is nearly separable. In this paper, we show how to make these algorithms efficient for data matrices that have many more rows than columns, so-called "tall-and-skinny matrices". One key component to these improved met…
LOFT separates subspace rotation and transformation for orthogonal fine-tuning.
problem Conflating subspace rotation and transformation in orthogonal fine-tuning.
method LOFT explicitly separates subspace rotation and transformation, using task-aware support selection.
result LOFT recovers principal-subspace orthogonal adaptation and improves efficiency-performance trade-off.
Efficient trainable front-end for neural speech enhancement.
problem Inefficient STFT front-ends in neural speech enhancement models.
method Butterfly mechanism for Fast Fourier Transform, trainable STFT window.
result Accuracy and efficiency improvements for low-compute systems.
The Stone-Weierstrass theorem aids in solving inverse problems on specific manifolds.
problem Inverse problems on holomorphically separable Kähler manifolds and conformally transversally anisotropic manifolds.
method Application of the Stone-Weierstrass theorem to show uniqueness in inverse problems.
result Generalization and simplification of earlier results in inverse problems.
Transformers cluster meaningless words around leaders for sentiment analysis.
problem Capturing context in sentiment analysis using transformers.
method Characterized transformers with hardmax self-attention and normalization, showing asymptotic convergence to clustered equilibrium.
result Transformers can effectively capture context by clustering meaningless words around leader words.
A low-rank transformation learning framework for subspace clustering and classification is here proposed. Many high-dimensional data, such as face images and motion sequences, approximately lie in a union of low-dimensional subspaces. The corresponding subspace clustering problem has been extensively studied in the lit…
ALT transforms time series data for better classification.
problem Efficiently classifying time series data with varying temporal scales.
method ALT algorithm using variable-length shifted time windows.
result State-of-the-art performance with minimal computational overhead.
Non-negative blind source separation (non-negative BSS), which is also referred to as non-negative matrix factorization (NMF), is a very active field in domains as different as astrophysics, audio processing or biomedical signal processing. In this context, the efficient retrieval of the sources requires the use of sig…
A novel graph spectral method for mixed categorical and numerical data.
problem Feature learning for mixed data types (numerical and categorical).
method Graph spectral decomposition of the graph Laplacian to model probabilistic dependence structure.
result Increased separability and clusterability of observations in the transformed feature space.
Randomly initialized neural networks can linearly separate arbitrary sets.
problem Mapping two arbitrary sets to linearly separable sets.
method Randomly initialized one-layer neural networks with sufficient width.
result With high probability, these networks can transform two sets into linearly separable sets.
An audiovisual speaker conversion method is presented for simultaneously transforming the facial expressions and voice of a source speaker into those of a target speaker. Transforming the facial and acoustic features together makes it possible for the converted voice and facial expressions to be highly correlated and f…
Deep networks achieve linear separability through progressive folding of data in higher dimensions.
problem How feed-forward networks achieve linear separability for classification tasks.
method Progressive folding of the data manifold in unoccupied higher dimensions.
result The folding operation allows efficient solutions by providing access to arbitrary regions in the distribution.
The paper proposes a method to identify power system oscillation modes using blind source separation.
problem Accurately identifying oscillation modes in power systems with renewable energy sources.
method A high-order blind source identification (HOBI) algorithm based on copula statistic combined with Hilbert transform and iteration procedure.
result The method can identify all oscillation modes and model order from a single channel of observation signals, outperforming state-of-the-art methods.
Recently, we proposed to transform the outputs of each hidden neuron in a multi-layer perceptron network to have zero output and zero slope on average, and use separate shortcut connections to model the linear dependencies instead. We continue the work by firstly introducing a third transformation to normalize the scal…
Models for audio source separation usually operate on the magnitude spectrum, which ignores phase information and makes separation performance dependant on hyper-parameters for the spectral front-end. Therefore, we investigate end-to-end source separation in the time-domain, which allows modelling phase information and…