arXiv research

Paper explores statistical and computational limits of estimating low-rank Gaussian mixtures.

We analyze computational limits of modern Hopfield models based on pattern norms.

New phases identified in neural scaling laws with compute limits.

Simple iterative method reduces deep network size significantly.

SGD learns sparse parities near computational limits with discontinuous phase transitions.

In this paper, we propose a general framework for tensor singular value decomposition (tensor SVD), which focuses on the methodology and theory for extracting the hidden low-rank structure from high-dimensional tensor data. Comprehensive results are developed on both the statistical and computational limits for tensor …

Paper shows hard computational limits for invariant causal prediction.

We analyze the computational limits of LoRA for transformer models using fine-grained complexity theory.

Study reveals limits of detecting local geometry in random graphs.

New method clusters tensors with heteroskedastic noise.

Computational limitations require more model parameters for robust learning.

Estimation under missing data shows computational and statistical limits for Gaussian data.

Paper explores limits of high-order clustering with planted structures.

Optimal multiscale learning of linear operators

Map matching of the GPS trajectory serves the purpose of recovering the original route on a road network from a sequence of noisy GPS observations. It is a fundamental technique to many Location Based Services. However, map matching of a low sampling rate on urban road network is still a challenging task. In this paper…

In this paper, we develop a framework to obtain graph abstractions for decision-making by an agent where the abstractions emerge as a function of the agent's limited computational resources. We discuss the connection of the proposed approach with information-theoretic signal compression, and formulate a novel optimizat…

Consider a noisy linear observation model with an unknown permutation, based on observing $y = Π^* A x^* + w$, where $x^* \in \mathbb{R}^d$ is an unknown vector, $Π^*$ is an unknown $n \times n$ permutation matrix, and $w \in \mathbb{R}^n$ is additive Gaussian noise. We analyze the problem of permutation recovery in a …

The paper classifies and computes limits of equivariant compactifications of groups.

Two-layer networks struggle with high frequencies due to numerical and computational limitations.

Algorithms for equilibrium computation generally make no attempt to ensure that the computed strategies are understandable by humans. For instance the strategies for the strongest poker agents are represented as massive binary files. In many situations, we would like to compute strategies that can actually be implement…

Deep learning's success requires vast computing power, making future progress unsustainable.

We describe stochastic Newton and stochastic quasi-Newton approaches to efficiently solve large linear least-squares problems where the very large data sets present a significant computational burden (e.g., the size may exceed computer memory or data are collected in real-time). In our proposed framework, stochasticity…

MEC-IP uses IP to efficiently find MECs in BNs from observational data.

In recent years, the spectral analysis of appropriately defined kernel matrices has emerged as a principled way to extract the low-dimensional structure often prevalent in high-dimensional data. Here we provide an introduction to spectral methods for linear and nonlinear dimension reduction, emphasizing ways to overcom…

Deep learning has been widely accepted as a promising solution for medical image segmentation, given a sufficiently large representative dataset of images with corresponding annotations. With ever increasing amounts of annotated medical datasets, it is infeasible to train a learning method always with all data from scr…

Learning an embedding for a large collection of items is a popular approach to overcome the computational limitations associated to one-hot encodings. The aim of item embedding is to learn a low dimensional space for the representations, able to capture with its geometry relevant features or relationships for the data …

Paper improves bootstrapping for off-policy reinforcement learning inference.

In this semi-tutorial paper, we first review the information-theoretic approach to account for the computational costs incurred during the search for optimal actions in a sequential decision-making problem. The traditional (MDP) framework ignores computational limitations while searching for optimal policies, essential…

Minimalist softmax attention learns constrained Boolean functions with supervision.

Why are classifiers in high dimension vulnerable to "adversarial" perturbations? We show that it is likely not due to information theoretic limitations, but rather it could be due to computational constraints. First we prove that, for a broad set of classification tasks, the mere existence of a robust classifier implie…

The study examines how verifier imperfections impact test-time scaling techniques.

Compressing neural nets is an active research problem, given the large size of state-of-the-art nets for tasks such as object recognition, and the computational limits imposed by mobile devices. We give a general formulation of model compression as constrained optimization. This includes many types of compression: quan…

Study reviews machine learning techniques for stress monitoring.

New method learns diverse protein scaffolds for motif design.

Develops methods to estimate high rank tensors from noisy data.

This paper speeds up kernel methods using sparsified Gaussian sketches.

Models applied on real time response task, like click-through rate (CTR) prediction model, require high accuracy and rigorous response time. Therefore, top-performing deep models of high depth and complexity are not well suited for these applications with the limitations on the inference time. In order to further impro…

FLEX optimizes exploration for nonlinear systems with minimal data.

Low-rank training improves neural network training on edge devices with non-volatile memory.

Transfer learning, in which a network is trained on one task and re-purposed on another, is often used to produce neural network classifiers when data is scarce or full-scale training is too costly. When the goal is to produce a model that is not only accurate but also adversarially robust, data scarcity and computatio…

The softmax content-based attention mechanism has proven to be very beneficial in many applications of recurrent neural networks. Nevertheless it suffers from two major computational limitations. First, its computations for an attention lookup scale linearly in the size of the attended sequence. Second, it does not enc…

Kernel methods represent one of the most powerful tools in machine learning to tackle problems expressed in terms of function values and derivatives due to their capability to represent and model complex relations. While these methods show good versatility, they are computationally intensive and have poor scalability t…

Understanding the relationship between the structure of light-harvesting systems and their excitation energy transfer properties is of fundamental importance in many applications including the development of next generation photovoltaics. Natural light harvesting in photosynthesis shows remarkable excitation energy tra…

We consider the optimization of a quadratic objective function whose gradients are only accessible through a stochastic oracle that returns the gradient at any given point plus a zero-mean finite variance random error. We present the first algorithm that achieves jointly the optimal prediction error rates for least-squ…

This paper analyzes and improves convergence in federated learning with biased client selection.

Develops methods to create consistent surrogate models for agent-based simulators.

Paper proposes efficient methods for high-order clustering in tensor block models.

A new Fusion method combines multiple distributions efficiently.