Automated spectral clustering algorithm discovers clusters without parameter tuning.
problem Automatic spectral clustering for multi-scale data.
method Heuristic iterative eigengap search with global and local scaling.
result Discover different patterns with accuracy >90% in most cases.
We develop an improved bound for the approximation error of the Nyström method under the assumption that there is a large eigengap in the spectrum of kernel matrix. This is based on the empirical observation that the eigengap has a significant impact on the approximation error of the Nyström method. Our approach is bas…
We consider the problem of principal component analysis (PCA) in a streaming stochastic setting, where our goal is to find a direction of approximate maximal variance, based on a stream of i.i.d. data points in Rd. A simple and computationally cheap algorithm for this is stochastic gradient descent (SGD), which…
A new GCN variant tackles large eigengaps in dense graphs and hypergraphs.
problem Large eigengaps in dense graphs and hypergraphs hinder popular GCN architectures.
method Uses pseudoinverse of the Laplacian and low-rank approximation for efficient computation.
result Improves runtime and accuracy in various experiments with real-world datasets.
We identify spectral conditions for reliable neural probe interpretation.
problem Unreliable performance of linear probes in interpreting neural representations.
method Formalized Spectral Identifiability Principle (SIP) based on eigengap and Fisher error.
result Reliability of neural probes depends on the eigengap relative to Fisher estimation error.
We study k-GenEV, the problem of finding the top k generalized eigenvectors, and k-CCA, the problem of finding the top k vectors in canonical-correlation analysis. We propose algorithms LazyEV and LazyCCA to solve the two problems with running times linearly dependent on the input size and…
This paper speeds up spectral clustering for large graphs by dilating their eigenspectrum.
problem Slow convergence in spectral clustering due to small eigengaps in graph Laplacians.
method Polynomial approximations to matrix operations that dilate the spectrum without changing eigenvectors.
result Significant acceleration of convergence in spectral clustering.
We study streaming principal component analysis (PCA), that is to find, in O(dk) space, the top k eigenvectors of a d×d hidden matrix Σ with online vectors drawn from covariance matrix Σ. We provide global convergence for Oja's algorithm which is popularly used in practice but lacks t…
New asynchronous algorithms improve speed in decentralized optimization networks.
problem Hard convergence analysis for asynchronous decentralized optimization.
method Continuized framework to analyze heterogeneous delays in event-driven updates.
result Achieves asynchronous speedup with convergence rate controlled by eigengap weighted by local delays.
This paper tackles distributed estimation of the top-L eigenspace in PCA for large data sets.
problem Challenges in estimating the top-L eigenspace in principal component analysis for large data sets.
method Proposes a novel multi-round algorithm using shift-and-invert preconditioning and convex optimization.
result Achieves a fast convergence rate and covers the targeted top-L eigenspace without explicit eigengap assumption.
Study spectral learning for odeco tensors, addressing initialization bottlenecks.
problem Recovering orthogonally decomposable tensors under noise.
method Investigates perturbation bounds, non-convex optimization, and initialization strategies.
result Initialization is the main bottleneck for efficient algorithms.
Algorithm estimates principal eigenvector with adaptive sensing, improving over non-adaptive methods.
problem Estimating principal eigenvector with limited scalar measurements.
method Compressed variant of Oja's algorithm using two adaptive measurements per sample.
result Convergence rate of O(λ1λ2d2/(Δ2t)) after t iterations, matching information-theoretic lower bound. This paper characterizes and explains the disagreement between two graph embedding methods.
problem Understanding why two popular graph embedding methods produce different results.
method End-to-end analysis of ASE-LSE latent subspaces, proving conditions for agreement and disagreement.
result No maximal-disagreement graph exists; disagreement is strictly below its theoretical ceiling.
In this paper, we propose an acceleration scheme for online memory-limited PCA methods. Our scheme converges to the first k>1 eigenvectors in a single data pass. We provide empirical convergence results of our scheme based on the spiked covariance model. Our scheme does not require any predefined parameters such as t…
We consider the minimization of a function defined on a Riemannian manifold M accessible only through unbiased estimates of its gradients. We develop a geometric framework to transform a sequence of slowly converging iterates generated from stochastic gradient descent (SGD) on M to an averaged i…
Tensor factorization arises in many machine learning applications, such knowledge base modeling and parameter estimation in latent variable models. However, numerical methods for tensor factorization have not reached the level of maturity of matrix factorization methods. In this paper, we propose a new method for CP te…
We consider the problem of clustering with the longest-leg path distance (LLPD) metric, which is informative for elongated and irregularly shaped clusters. We prove finite-sample guarantees on the performance of clustering with respect to this metric when random samples are drawn from multiple intrinsically low-dimensi…
Spectral clustering is one of the most widely used techniques for extracting the underlying global structure of a data set. Compressed sensing and matrix completion have emerged as prevailing methods for efficiently recovering sparse and partially observed signals respectively. We combine the distance preserving measur…
In this paper, we determine the optimal convergence rates for strongly convex and smooth distributed optimization in two settings: centralized and decentralized communications over a network. For centralized (i.e. master/slave) algorithms, we show that distributing Nesterov's accelerated gradient descent is optimal and…
A new language for neural architecture search decouples search spaces and algorithms.
problem Current neural architecture search methods are limited to specific use-cases and lack general-purpose constructs.
method Proposes a formal language for encoding search spaces over general computational graphs, allowing modular, composable, and reusable encodings.
result The language enables easy experimentation with different search spaces and algorithms without reinventing the wheel.
This paper compares Grid Search, Random Search, and Genetic Algorithm for NAS.
problem Hyperparameter optimization for neural architecture search.
method Comparison of Grid Search, Random Search, and Genetic Algorithm.
result Genetic Algorithm outperforms Grid Search and Random Search in terms of accuracy and execution time.
We consider the problem of detecting a tight community in a sparse random network. This is formalized as testing for the existence of a dense random subgraph in a random graph. Under the null hypothesis, the graph is a realization of an Erdös-Rényi graph on N vertices and with connection probability p0; under the …
Efficient search methods can outperform random search on challenging tasks.
problem Comparing the performance of efficient and random search methods in neural architecture search.
method Comparison of weight sharing and random search methods on progressively larger search spaces for image classification and detection.
result Efficient search methods can provide substantial gains over random search on large, realistic tasks.
A new framework generates large hierarchical search spaces for neural architectures.
problem Discovering neural architectures from simple blocks is hard.
method Context-free grammars for a unified, scalable search space.
result Efficiently searches over complete architectures, outperforming existing methods.
MICO uses mutual information co-training to improve selective search efficiency.
problem Efficiently search and route unseen queries in large-scale search systems.
method Mutual Information Co-training framework for selective search with minimal supervision.
result Significantly improves performance on multiple metrics of selective search.
This work recommends personalized search stories to users based on their interests.
problem Personalized search story recommendation within search engines.
method Deep reinforcement learning architecture trained by imitation learning and reinforcement learning.
result Empirically demonstrated effectiveness on real-world data sets.
A method to reduce memory usage in NAS by pruning the search space.
problem High GPU memory consumption in One-Shot NAS techniques.
method Utilising Zero-Shot NAS to prune the search space before applying One-Shot NAS.
result Reduces memory consumption by 81% while maintaining accuracy.
Paper uses CMAB to improve NAS efficiency and accuracy.
problem Improving efficiency and accuracy of NAS for DNNs.
method Formulated NAS as CMAB, used Nested Monte-Carlo Search.
result Discovered cell structure achieves comparable accuracy to state-of-the-art, 20x faster.
New agent learns from previous search spaces to improve NAS efficiency.
problem NAS requires restarting learning from scratch between different search spaces.
method Transformer-based agent for joint training and knowledge transfer.
result Efficient knowledge transfer between search spaces improves NAS performance.
PGS uses neural networks to improve policies online without search trees.
problem Limited scalability of Monte Carlo Tree Search (MCTS) for high branching factor games.
method Adapts a neural network simulation policy via policy gradient updates, avoiding search trees.
result PGS achieves comparable performance to MCTS and defeats strong Hex agents.
New method reduces NAS search time and complexity.
problem High computational cost and complexity in NAS.
method Differentiable search space with annealing and pruning.
result Achieves 1.68% error on CIFAR-10 with 0.2 GPU days.
Approaches to learning Bayesian networks from data typically combine a scoring function with a heuristic search procedure. Given a Bayesian network structure, many of the scoring functions derived in the literature return a score for the entire equivalence class to which the structure belongs. When using such a scoring…
FiGS searches over a larger space of architectures for efficient mobile models.
problem Designing small, efficient deep networks for mobile devices.
method Differentiable search method using sparse regularization and Logistic-Sigmoid distribution.
result FiGS produces state-of-the-art parameter-efficient models on ImageNet and improves object detection performance.
This work proposes searching for optimal operation distribution in neural architecture search.
problem Finding optimal neural architecture with specific operations and connections.
method Search for the optimal operation distribution, providing a stochastic and approximate solution.
result Operation distribution holds enough discriminating power to reliably identify a solution and is easier to optimise than traditional encodings.
CrossBeam learns to search more efficiently in program synthesis.
problem Efficiently searching through vast program spaces.
method Trains a neural model to guide program synthesis, combining previously explored programs.
result CrossBeam explores much smaller portions of the program space compared to state-of-the-art methods.
Deep-n-Cheap automates deep learning model search for low complexity.
problem Finding efficient deep learning models for various datasets.
method Automated search framework for architecture and hyperparameters, including search transfer.
result Models offer comparable performance to state-of-the-art but are faster to train.
Paper presents a new trie for integer sketches to improve similarity searches.
problem Efficient similarity searches on integer sketches.
method Introduces a novel b-bit sketch trie that leverages succinct data structures. result Significantly improves search time and space-efficiency of similarity searches.
Improves diffusion model performance and efficiency through classical search.
problem Tackles inference-time control in diffusion models.
method Proposes a framework combining local and global search for efficient navigation.
result Significant gains in performance and efficiency across various domains.
Paper proposes a new evaluation method for NAS search phase.
problem NAS search phase effectiveness not well evaluated.
method Compare NAS solutions with random selection; evaluate weight sharing strategy.
result State-of-the-art NAS algorithms perform similarly to random selection.
Random search outperforms leading NAS methods in neural architecture search.
problem Improving the reproducibility and efficiency of neural architecture search.
method Proposed random search with early-stopping and weight-sharing as NAS baselines.
result Random search with weight-sharing achieves state-of-the-art results on PTB and highly competitive results on CIFAR-10.
einspace expands NAS search space to include diverse neural architectures.
problem NAS results are often limited to existing structures; new designs are rare.
method einspace uses a probabilistic context-free grammar to create a versatile search space.
result einspace discovers novel and improved architectures, including convolutions and attention.
Paper improves full-text search engines for fast exact NNS in binary codes.
problem Efficient nearest neighbor search in Hamming space for full-text search engines.
method Revisits and combines three techniques from information retrieval: bit operation, subs-code filtering, and data preprocessing with permutation.
result Significant speed-ups for NNS in binary codes over state-of-the-art term match approach.
We consider a framework for structured prediction based on search in the space of complete structured outputs. Given a structured input, an output is produced by running a time-bounded search procedure guided by a learned cost function, and then returning the least cost output uncovered during the search. This framewor…
Method predicts disease outbreaks using search logs, overcoming instability.
problem Predicting disease outbreaks from search logs is challenging due to short-term and long-term instability.
method Seasonal-adjustment method decomposes logs into seasonal, trend, and irregular components; feature selection method selects relevant search terms.
result Proposed method outperforms comparative methods in prediction accuracy for seven of ten diseases.
Paper develops DP methods for low-rank matrix estimation with near-optimal performance.
problem Estimating a low-rank matrix under differential privacy constraints.
method Introduced computationally efficient DP-initialization and Riemannian optimization-based DP-RGrad algorithm.
result DP-RGrad achieves near-optimal convergence rate under weak differential privacy constraints.
SGAS improves neural architecture search by choosing and pruning operations greedily.
problem NAS often fails to generalize in final evaluation.
method Divides search into sub-problems and chooses/prunes candidate operations greedily.
result SGAS finds state-of-the-art architectures with minimal computational cost.
UNAS combines DNAS and RL for efficient architecture search.
problem Discovering high accuracy or low latency neural architectures.
method Unified framework combining differentiable and reinforcement learning approaches.
result UNAS achieves state-of-the-art accuracy on CIFAR-10, CIFAR-100, and ImageNet datasets.
Efficient neural architecture search by sampling structure and operations.
problem Efficiently searching for optimal neural architectures.
method Decouples structure and operation search, using reinforcement learning with policy vectors.
result Significantly improved efficiency compared to traditional methods.