Paper proves hardness of learning various complex models under local pseudorandom generators.
problem Hardness of learning various complex models.
method Existence of local pseudorandom generators.
result Proves hardness of learning shallow ReLU neural networks and other models.
Hard to estimate L2-accurate scores without strong assumptions.
problem Estimating the score of unknown data distributions accurately.
method Reduction to generating samples and leveraging lattice-based cryptography hardness.
result Score estimation is computationally hard even with polynomial sample complexity.
Hardness proof for agnostically learning halfspaces from worst-case lattice problems.
problem Agnostically learning halfspaces in the presence of noise.
method Reduction to worst-case lattice problems (GapSVP, SIVP).
result No efficient algorithm can achieve misclassification error better than 1/2 - γ under given hardness assumptions.
New findings show learning deeper neural networks is hard even with Gaussian inputs and non-degenerate weights.
problem The computational complexity of learning neural networks, especially deeper ones.
method Smoothed analysis framework and local pseudorandom generators.
result Learning depth-3 ReLU networks under Gaussian input distribution is hard even if weight matrices are non-degenerate.
Study reward-free RL in non-linear settings, improving efficiency and removing assumptions.
problem Improving sample efficiency in reward-free reinforcement learning for non-linear function approximation.
method Proposed RFOLIVE algorithm for minimal structural assumptions, analyzed hardness results for reward-free and reward-aware exploration.
result Statistical efficiency and hardness results under various structural assumptions, no need for reachability or explorability assumptions.
Paper investigates hardness of learning neural networks under manifold hypothesis.
problem Hardness of learning neural networks under the manifold hypothesis.
method Extending proofs of hardness in the SQ and cryptographic settings to the geometric setting.
result Learning is hard under input manifolds of bounded curvature but learnable with additional assumptions on manifold volume.
Hardness proven for learning neural networks with polynomial size and Gaussian inputs.
problem Learning one hidden layer ReLU neural networks with polynomial size and Gaussian inputs.
method Based on the hardness of the Continuous Learning with Errors (CLWE) problem.
result Hardness of learning neural networks is proven under standard cryptographic assumptions.
We give a reduction from {\sc clique} to establish that sparse PCA is NP-hard. The reduction has a gap which we use to exclude an FPTAS for sparse PCA (unless P=NP). Under weaker complexity assumptions, we also exclude polynomial constant-factor approximation algorithms.
New algorithm for planning in observable POMDPs in quasi-polynomial time.
problem Planning in POMDPs is computationally hard.
method Assumption of well-separated distributions on states and observations leads to quasi-succinct descriptions of near-optimal policies.
result Quasipolynomial-time algorithm for planning in observable POMDPs.
Reduces learning periodic neural networks to lattice problems, proving hardness under cryptographic assumptions.
problem Learning single periodic neurons in noisy environments.
method Reduction to worst-case lattice problems, using LLL algorithm.
result Polynomial-time algorithms for learning these functions are hard under cryptographic assumptions.
Bailouts in financial networks are hard to optimize due to NP-hardness.
problem Optimizing bailouts in a network of insolvent banks.
method Modeling bailouts as an optimization problem, proving NP-hardness and inapproximability.
result Banks can strategically alter debt contracts to increase their market value in the event of a bailout.
New lower bounds show learning intersections of halfspaces is hard even for a few halfspaces.
problem Learning intersections of halfspaces in polynomial time under standard assumptions.
method Unified connection to parallel pancakes distribution for proving hardness.
result Learning ω(loglogN) halfspaces in dimension N requires super-polynomial time under standard assumptions. We study --both in theory and practice-- the use of momentum motions in classic iterative hard thresholding (IHT) methods. By simply modifying plain IHT, we investigate its convergence behavior on convex optimization criteria with non-convex constraints, under standard assumptions. In diverse scenaria, we observe that …
Paper relaxes faithfulness assumption for causal discovery using interventions.
problem Violation of faithfulness assumption in natural systems leads to incorrect causal structure identification.
method Use intervention-immediacy faithfulness assumption to identify causal structures with hard interventions.
result Interventions contain information about causal structure that can identify causal structures when faithfulness is violated.
Introduces a continuous version of LWE problem.
problem Hardness of learning mixtures of Gaussians.
method Polynomial-time quantum reduction from CLWE to lattice problems.
result CLWE shares hardness with LWE.
Study limits of testing algorithms without assumptions, finding key performance bounds.
problem Understanding algorithm performance without making assumptions.
method Exploring fundamental limits of algorithm performance testing.
result Hardness results for evaluating algorithm performance and comparing multiple algorithms.
New algorithm learns POMDPs without computational oracles.
problem Learning near-optimal policies in POMDPs with computationally hard oracles.
method Quasipolynomial-time algorithm using barycentric spanners for policy covers.
result First oracle-free learning algorithm for observable POMDPs.
New research shows deep learning struggles with hard problems due to biased data generation.
problem Deep learning's limitations in solving computationally hard problems.
method Proved that polynomial-time sample generators for NP-hard problems sample from easier sub-problems.
result Machine learning models trained on biased datasets overestimate their accuracy for hard problems.
Noise makes learning linear thresholds hard, but algorithms can still learn near-optimal thresholds.
problem Learning linear thresholds in noisy data.
method Exploiting natural assumptions on data-generating process.
result Efficient learning of near-optimal linear thresholds is still possible with small data even in the presence of noise.
Boolean logic used for neural network training and inference, with convergence analysis.
problem Discrete optimization in neural networks with Boolean logic.
method Boolean logic backpropagation with convergence analysis.
result First convergence analysis for Boolean logic in neural networks.
WEINCE improves contrastive learning by correcting softmax biases.
problem Softmax in InfoNCE can lead to misaligned statistical assumptions in contrastive learning.
method WEINCE uses anchor-wise online batch statistics to blend softmax logits with an endpoint shortfall correction.
result WEINCE yields consistent improvements in frozen-feature evaluation across five vision benchmarks.
Quantum computers outperform classical methods in density modeling.
problem Density modeling with quantum computers.
method Quantum-classical separation for density modeling.
result Quantum computers offer a super-polynomial advantage over classical algorithms for density modeling.
Empirical risk minimization (ERM) is ubiquitous in machine learning and underlies most supervised learning methods. While there has been a large body of work on algorithms for various ERM problems, the exact computational complexity of ERM is still not understood. We address this issue for multiple popular ERM problems…
Proposes new random models for fuzzy clustering similarity measures.
problem Challenges in choosing a random model for fuzzy clustering similarity measures.
method Introduces three intuitive and explainable random models for fuzzy clusterings.
result Each random model has distinct behavior, emphasizing the importance of accurate model selection.
Sparse linear regression is hard to solve efficiently, even with k-sparse solutions.
problem Sparse linear regression problem with k-sparse solutions.
method Fine-grained complexity and hardness assumptions.
result No better-than-brute-force algorithms exist for sparse linear regression.
HARFE approximates sparse additive functions using random features and ridge regression.
problem Approximating high-dimensional sparse additive functions.
method Hard-ridge random feature expansion with sparse ridge regression and hard-thresholding pursuit.
result HARFE method converges with a given error bound and achieves lower error than other algorithms.
Recovery of low-rank matrices from a small number of linear measurements is now well-known to be possible under various model assumptions on the measurements. Such results demonstrate robustness and are backed with provable theoretical guarantees. However, extensions to tensor recovery have only recently began to be st…
The restricted isometry property (RIP) for design matrices gives guarantees for optimal recovery in sparse linear models. It is of high interest in compressed sensing and statistical learning. This property is particularly important for computationally efficient recovery methods. As a consequence, even though it is in …
We present a new anytime algorithm that achieves near-optimal regret for any instance of finite stochastic partial monitoring. In particular, the new algorithm achieves the minimax regret, within logarithmic factors, for both "easy" and "hard" problems. For easy problems, it additionally achieves logarithmic individual…
Study on computing and estimating calibration distance, showing hardness and efficiency.
problem Computing and estimating calibration distance under different assumptions.
method Efficient algorithm for exact computation, polynomial-time approximation scheme; sample-based estimation for upper bounds.
result The problem becomes NP-hard when assumptions are removed, but efficient algorithms exist under certain conditions.
Paper presents a randomized algorithm for SPCA with high probability approximation.
problem Sparse Principal Component Analysis (SPCA) is NP-hard.
method Based on basic SDP relaxation, the algorithm constructs deterministic and randomized solutions.
result The algorithm achieves an approximation ratio of at most the sparsity constant with high probability.
It is becoming increasingly important to understand the vulnerability of machine learning models to adversarial attacks. In this paper we study the feasibility of robust learning from the perspective of computational learning theory, considering both sample and computational complexity. In particular, our definition of…
In our recent work (Bubeck, Price, Razenshteyn, arXiv:1805.10204) we argued that adversarial examples in machine learning might be due to an inherent computational hardness of the problem. More precisely, we constructed a binary classification task for which (i) a robust classifier exists; yet no non-trivial accuracy c…
Studying general quantum many-body systems is one of the major challenges in modern physics because it requires an amount of computational resources that scales exponentially with the size of the system.Simulating the evolution of a state, or even storing its description, rapidly becomes intractable for exact classical…
The estimation of an f-divergence between two probability distributions based on samples is a fundamental problem in statistics and machine learning. Most works study this problem under very weak assumptions, in which case it is provably hard. We consider the case of stronger structural assumptions that are commonly sa…
New insights link diverse statistical problems via secret leakage planted clique.
problem Statistical-computational gaps in inference problems.
method Secret leakage planted clique as a new hardness assumption for reductions.
result Establishes tight statistical-computational tradeoffs for various problems.
New algorithm robustly estimates sparse models in high dimensions with corrupted data.
problem Estimating latent variable models with arbitrarily corrupted samples in high dimensional space.
method Trimmed (Gradient) Expectation Maximization with trimming gradients and hard thresholding steps.
result The algorithm converges to near optimal statistical rate geometrically under certain conditions.
Tensor models improve joint EEG and fMRI analysis.
problem Jointly analyzing EEG and fMRI for brain function studies.
method Soft and flexible coupling of tensor decompositions for EEG and fMRI.
result Tensorial methods outperform ICA in multi-modal analysis.
New results connect RBMs to neural networks, improving learning efficiency.
problem Learning graphical models with latent variables is difficult.
method New connections to learning two-layer neural networks under ℓ∞ bounded input. result Improved algorithm for learning supervised RBMs.
Hard to approximate critical points for simple nonconvex functions.
problem Approximating critical points of nonconvex functions.
method Proving hardness results for polynomial-time approximation of critical points.
result Proving that approximating critical points is intractable for simple nonconvex functions.
Although neural networks are routinely and successfully trained in practice using simple gradient-based methods, most existing theoretical results are negative, showing that learning such networks is difficult, in a worst-case sense over all data distributions. In this paper, we take a more nuanced view, and consider w…
Learning a causal effect from observational data is not straightforward, as this is not possible without further assumptions. If hidden common causes between treatment X and outcome Y cannot be blocked by other measurements, one possibility is to use an instrumental variable. In principle, it is possible under some…
Tyler's M-estimator's phase transition at DS-SNR = 1 is resolved.
problem Robust Subspace Recovery
method Tyler's M-estimator
result TME converges exactly to the true subspace for DS-SNR >= 1 under a new stability condition.
New algorithms minimize regret with multiple best arms in large action spaces.
problem Minimizing regret in multi-armed bandit with multiple best arms.
method Adaptive algorithms that automatically adapt to hardness level, with theoretical regret bounds and lower bounds.
result Proposed algorithms achieve optimal or near-optimal performance, depending on additional knowledge.
This paper defends SVMs against poisoning attacks using DBSCAN and hardness proofs.
problem Adversarial injection of specially crafted samples into training data to misclassify SVMs.
method Two strategies: robust SVM algorithms and data sanitization (DBSCAN).
result Proves hardness of simple SVM problem and effectiveness of DBSCAN for poisoning attacks.
New algorithm learns halfspaces over hypercube with random bit flips.
problem Agnostic learning of Boolean halfspaces over discrete domains is computationally hard.
method Smoothed analysis with random bit flips for discrete inputs.
result First efficient algorithm for smoothed agnostic learning of halfspaces over Boolean hypercube.
Over recent years, devising classification algorithms that are robust to adversarial perturbations has emerged as a challenging problem. In particular, deep neural nets (DNNs) seem to be susceptible to small imperceptible changes over test instances. However, the line of work in provable robustness, so far, has been fo…
Study learning from multiple thinkers providing step-by-step solutions to problems.
problem Learning from multiple, possibly different, thinkers providing step-by-step solutions to problems.
method Active learning algorithm that uses CoT data from multiple thinkers and end-result data.
result Learning can be hard from CoT supervision provided by two or a few different thinkers, but a generic algorithm can learn efficiently.