Research
On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

168,742 papers · 148 categories

Trend · papers per month

70139209278 · Jun 202019922001200920172026
48 results for clustered attention

Clustered attention improves transformer efficiency for large sequences.

problem Quadratic complexity of transformer attention matrix for large sequences.
method Group queries into clusters, compute attention only for centroids, and use centroids to approximate key/query dot products.
result Linear complexity with respect to sequence length for a fixed number of clusters.

POTA improves short text clustering by generating reliable pseudo-labels.

problem Limited discriminative representations in short texts.
method POTA uses instance-level attention and optimal transport for semantic consistency and cluster structure.
result POTA outperforms state-of-the-art methods in short text clustering.

Transformers can cluster data from Gaussian mixtures without supervision.

problem Clustering data from Gaussian mixtures without labeled data.
method Theoretical analysis of attention-based layers, focusing on a simplified two-head attention layer and an identity matrix attention layer.
result Attention-based layers can align with true mixture centroids and adapt to input-specific distributions.

Centroid Transformers reduce memory and computation by summarizing inputs into centroids.

problem Efficiently summarize inputs with reduced memory and computation.
method Generalizes self-attention to map N inputs to M centroids (M ≤ N), reducing complexity.
result Centroid Transformers reduce memory and computation while preserving key information.

Transformers cluster meaningless words around leaders for sentiment analysis.

problem Capturing context in sentiment analysis using transformers.
method Characterized transformers with hardmax self-attention and normalization, showing asymptotic convergence to clustered equilibrium.
result Transformers can effectively capture context by clustering meaningless words around leader words.

Graph clustering is a fundamental task which discovers communities or groups in networks. Recent studies have mostly focused on developing deep learning approaches to learn a compact graph embedding, upon which classic clustering methods like k-means or spectral clustering algorithms are applied. These two-step framewo…

2019-06-15abs ↗pdf ↗

Particles representing tokens cluster in Transformers, influenced by initial tokens and matrix spectrum.

problem Understanding the geometry of learned representations in Transformers.
method Viewing Transformers as particle systems, applying dynamical systems and partial differential equations.
result Particles cluster towards limiting objects, confirming context-awareness and the emergence of leaders.

Clustering is one of the most important unsupervised problems in machine learning and statistics. Among many existing algorithms, kernel k-means has drawn much research attention due to its ability to find non-linear cluster boundaries and its inherent simplicity. There are two main approaches for kernel k-means: SVD o…

2016-06-06abs ↗pdf ↗

Paper proposes new methods for improving interatomic potentials.

problem Limitations of conventional SO(2) Linear architectures in MLIPs.
method Direct Cartesian construction, recursive Clebsch-Gordan construction, Edge Complex Product Basis, Radial Rotary Complex Attention.
result TECE-OAM-RRA-1.0 achieves SOTA performance on Matbench Discovery.

A panoply of multi-view clustering algorithms has been developed to deal with prevalent multi-view data. Among them, spectral clustering-based methods have drawn much attention and demonstrated promising results recently. Despite progress, there are still two fundamental questions that stay unanswered to date. First, h…

2019-09-16abs ↗pdf ↗

Paper improves short text clustering by integrating semantic relationships into Optimal Transport.

problem Erroneous pseudo-labels caused by neglecting semantic consistency in existing OT methods.
method Designs an instance-level attention mechanism to capture semantic relationships and integrates them into the OT formulation.
result Generates reliable pseudo-labels that improve clustering accuracy.

Proposes SDCN to integrate structural information into deep clustering.

problem Lack of attention to structural information in representation learning for clustering.
method Designs a delivery operator to transfer autoencoder representations to GCN layers and uses a dual self-supervised mechanism.
result SDCN consistently outperforms state-of-the-art techniques in clustering tasks.

Transformers learn to cluster Gaussian mixtures as well as the EM algorithm.

problem Learning guarantees of Transformers in multi-class clustering of Gaussian mixtures.
method Developed a theory connecting Transformer's Softmax Attention layers to the EM algorithm's workflow.
result Transformers achieve minimax optimal rate for clustering Gaussian mixtures with sufficient training samples and initialization.

Image clustering is an important but challenging task in machine learning. As in most image processing areas, the latest improvements came from models based on the deep learning approach. However, classical deep learning methods have problems to deal with spatial image transformations like scale and rotation. In this p…

2019-02-09abs ↗pdf ↗

Model-based clustering is a popular approach for clustering multivariate data which has seen applications in numerous fields. Nowadays, high-dimensional data are more and more common and the model-based clustering approach has adapted to deal with the increasing dimensionality. In particular, the development of variabl…

2017-07-02abs ↗pdf ↗

GAP learns node representations by attending to different parts of its neighborhood.

problem Context-free learning of node representations in graph representation learning.
method Graph Neighborhood Attentive Pooling (GAP) using attentive pooling networks.
result GAP outperforms 10 state-of-the-art methods on link prediction and clustering tasks.

The clustering methods have recently absorbed even-increasing attention in learning and vision. Deep clustering combines embedding and clustering together to obtain optimal embedding subspace for clustering, which can be more effective compared with conventional clustering methods. In this paper, we propose a joint lea…

2019-04-30abs ↗pdf ↗

Quantum computing for machine learning attracts increasing attention and recent technological developments suggest that especially adiabatic quantum computing may soon be of practical interest. In this paper, we therefore consider this paradigm and discuss how to adopt it to the problem of binary clustering. Numerical …

2017-06-17abs ↗pdf ↗

DEMVC improves multi-view clustering with collaborative training and deep autoencoders.

problem Existing multi-view clustering methods have high computation and space complexities or lack representation capability.
method DEMVC learns embedded representations of multiple views individually using deep autoencoders and collaboratively trains all views.
result DEMVC achieves significant improvements over state-of-the-art methods on multi-view datasets.

Convex clustering, a convex relaxation of k-means clustering and hierarchical clustering, has drawn recent attentions since it nicely addresses the instability issue of traditional nonconvex clustering methods. Although its computational and statistical properties have been recently studied, the performance of convex c…

2016-01-18abs ↗pdf ↗

Clustering on hypergraphs has been garnering increased attention with potential applications in network analysis, VLSI design and computer vision, among others. In this work, we generalize the framework of modularity maximization for clustering on hypergraphs. To this end, we introduce a hypergraph null model, analogou…

2018-12-28abs ↗pdf ↗

In this paper we take a problem of unsupervised nodes clustering on graphs and show how recent advances in attention models can be applied successfully in a "hard" regime of the problem. We propose an unsupervised algorithm that encodes Bethe Hessian embeddings by optimizing soft modularity loss and argue that our mode…

2019-05-20abs ↗pdf ↗

Enhances robustness of multi-view clustering via partition fusion.

problem Dealing with noises and inconsistency in multi-view data.
method Generates multiple partitions, integrates them, and co-evolves graph learning, partition generation, and view weight learning.
result Empirical results verify the effectiveness and robustness of the proposed approach.

Multi-view clustering has received much attention recently. Most of the existing multi-view clustering methods only focus on one-sided clustering. As the co-occurring data elements involve the counts of sample-feature co-occurrences, it is more efficient to conduct two-sided clustering along the samples and features si…

2019-05-25abs ↗pdf ↗

Perfect clustering achieved in hypergraphs with enough interactions.

problem Complexity and lack of tractable models for analyzing hypergraphs.
method Introduced an interaction hypergraph model for analyzing hypergraphs, defined latent embeddings, and analyzed spectral estimators.
result A spectral estimate of interaction latent positions can achieve perfect clustering with enough interactions.

Community detection, which focuses on clustering nodes or detecting communities in (mostly) a single network, is a problem of considerable practical interest and has received a great deal of attention in the research community. While being able to cluster within a network is important, there are emerging needs to be ab…

2016-06-08abs ↗pdf ↗

Consensus clustering fuses diverse basic partitions (i.e., clustering results obtained from conventional clustering methods) into an integrated one, which has attracted increasing attention in both academic and industrial areas due to its robust and effective performance. Tremendous research efforts have been made to t…

2019-05-31abs ↗pdf ↗

Interprets how intrinsic motivation shapes behavior in RL agents.

problem Understanding how intrinsic motivation influences behavior in reinforcement learning agents.
method Analyzed five RL agents in procedurally generated environments using various interpretability techniques.
result Curiosity-driven agents exhibit broader and more dynamic attention than extrinsically motivated agents.

A tutorial on various methods for clustering longitudinal data.

problem Identifying groups with different trends in longitudinal data.
method Group-based trajectory modeling, growth mixture modeling, longitudinal k-means.
result Strengths, limitations, and model extensions of the methods are discussed.

We consider the problem of community detection or clustering in the labeled Stochastic Block Model (LSBM) with a finite number KK of clusters of sizes linearly growing with the global population of items nn. Every pair of items is labeled independently at random, and label \ell appears with probability $p(i,j,\ell)…

2015-10-20abs ↗pdf ↗

This paper introduces GEMINI, a new metric for unsupervised neural network training that avoids the need for regularizations.

problem The mutual information (MI) as a clustering objective does not lead to satisfactory clusters.
method The authors generalised the mutual information by changing its core distance, introducing the Generalised Mutual Information (GEMINI).
result Some GEMINIs do not require regularizations when training and can automatically select the number of clusters.

Study shows how specialized attention circuits emerge during transformer training.

problem Understanding the mechanisms of transformer training dynamics at large scales.
method Controlled sparse modular addition task; monitoring token evolution via visual sandbox.
result Specialized attention circuits (clustering heads) naturally emerge during training.

This paper introduces GEMINI, a new mutual information metric for unsupervised neural network training.

problem The mutual information (MI) as a clustering objective does not lead to satisfactory clusters.
method The authors generalised MI by changing its core distance, introducing GEMINIs that do not require regularizations and can automatically select the number of clusters.
result GEMINIs can automatically select the number of clusters without requiring a priori knowledge of the number of clusters.

Multi-task clustering (MTC) has attracted a lot of research attentions in machine learning due to its ability in utilizing the relationship among different tasks. Despite the success of traditional MTC models, they are either easy to stuck into local optima, or sensitive to outliers and noisy data. To alleviate these p…

2018-08-24abs ↗pdf ↗