DC3 uses deep learning to solve hard-constrained optimization problems efficiently.
problem Hard constraints in optimization problems make classical solvers slow and infeasible.
method DC3 employs a differentiable procedure to enforce feasibility and unrolls corrections for inequality constraints.
result DC3 achieves near-optimal solutions while maintaining feasibility in both synthetic and real-world tasks.
Trading system uses NP-hard optimization to select stocks for high Sharpe ratio trading.
problem Finding profitable, uncorrelated stocks for high Sharpe ratio trading.
method NP-hard combinatorial optimization using Ising machine and simulated bifurcation algorithm.
result Trading strategy with FPGA-based system achieves 164 μs response latency.
In this thesis I explore challenging discrete energy minimization problems that arise mainly in the context of computer vision tasks. This work motivates the use of such "hard-to-optimize" non-submodular functionals, and proposes methods and algorithms to cope with the NP-hardness of their optimization. Consequently, t…
Bailouts in financial networks are hard to optimize due to NP-hardness.
problem Optimizing bailouts in a network of insolvent banks.
method Modeling bailouts as an optimization problem, proving NP-hardness and inapproximability.
result Banks can strategically alter debt contracts to increase their market value in the event of a bailout.
Hard visual attention is a promising approach to reduce the computational burden of modern computer vision methodologies. Hard attention mechanisms are typically non-differentiable. They can be trained with reinforcement learning but the high-variance training this entails hinders more widespread application. We show h…
New RGraSP framework for efficient non-convex optimization.
problem Large-scale non-convex sparsity-constrained optimization problems.
method Relaxed gradient support pursuit with semi-stochastic gradient hard thresholding.
result Our algorithms converge faster with lower per-iteration cost.
This paper shows how optimizing with hard negative examples improves image retrieval.
problem Training with hard negative examples leads to poor training behavior.
method Characterize the space of triplets, derive why hard negatives fail, and offer a fix to the loss function.
result Optimizing with hard negative examples leads to more generalizable features and better image retrieval.
We consider network sparsification as an L0-norm regularized binary optimization problem, where each unit of a neural network (e.g., weight, neuron, or channel, etc.) is attached with a stochastic binary gate, whose parameters are jointly optimized with original network parameters. The Augment-Reinforce-Merge (ARM),…
Study shows optimal RL with transition look-ahead is NP-hard for ℓ≥2.
problem Optimal reinforcement learning with transition look-ahead is computationally hard.
method Proved NP-hardness for ℓ≥2 using linear programming. result There is a precise boundary between tractable and intractable cases for RL with look-ahead.
MAML outperforms NAL in diverse task landscapes.
problem Understanding when and how MAML outperforms NAL in various task landscapes.
method Analytical and numerical studies in a linear regression setting with a mixture of easy and hard tasks.
result MAML gains over NAL when there is task hardness discrepancy and optimal solutions of hard tasks are closely packed.
Optimal CL requires perfect memory and is NP-hard.
problem Designing CL algorithms that perform reliably and avoid catastrophic forgetting.
method Theoretical approach to derive computational properties of optimal CL algorithms.
result Optimal CL algorithms generally solve an NP-hard problem and require perfect memory.
We consider stochastic gradient descent (SGD) for least-squares regression with potentially several passes over the data. While several passes have been widely reported to perform practically better in terms of predictive performance on unseen data, the existing theoretical analysis of SGD suggests that a single pass i…
GSO framework optimizes COPs on graphs using Gumbel-softmax.
problem Finding optimal configurations or network structures in combinatorial optimization problems.
method Introducing Gumbel-softmax technique for direct optimization of objective functions.
result High-quality solutions obtained with less time compared to traditional methods.
Transformers struggle to learn Markovian dynamics, showing NP-hard optimization challenges.
problem Understanding transformers' limitations in learning Markovian dynamical functions.
method Investigated through a structured ICL setup, analyzing loss landscapes and parameter optimization.
result Recovering optimal transformer parameters for Markovian functions is NP-hard.
Proves hardness of semi-discrete optimal transport and proposes regularization methods.
problem Computing Wasserstein distance between discrete and non-discrete probability measures.
method Proves hardness, introduces distributionally robust dual optimal transport, regularizes primal objective, uses stochastic gradient descent.
result Regularization schemes and improved convergence guarantees for semi-discrete optimal transport problems.
A new algorithm optimizes graph problems faster and more accurately.
problem Hard optimization problems on graphs.
method Gumbel-softmax technique with gradient descent and evolution strategy.
result High-quality solutions obtained with less time.
Iterative thresholding algorithms seek to optimize a differentiable objective function over a sparsity or rank constraint by alternating between gradient steps that reduce the objective, and thresholding steps that enforce the constraint. This work examines the choice of the thresholding operator, and asks whether it i…
Improved iterative hard thresholding for faster, sparser solutions.
problem Finding sparser solutions without sacrificing runtime.
method Adaptive regularization framework applied to iterative hard thresholding.
result Returns solutions with sparsity O(sκ), improving over existing methods. Optimal intervention in economic networks modeled as influence maximization, with hard computational problems.
problem Optimal intervention in economic networks modeled as influence maximization.
method Transformed into influence maximization-like form, with theoretical and practical implications.
result Optimal intervention is NP-hard and cannot be approximated to a constant factor in polynomial time.
We study the problem of attacking a machine learning model in the hard-label black-box setting, where no model information is revealed except that the attacker can make queries to probe the corresponding hard-label decisions. This is a very challenging problem since the direct extension of state-of-the-art white-box at…
HardNet adds hard constraints to neural networks without sacrificing performance.
problem Ensuring adherence to input-dependent constraints in neural networks.
method Appends a differentiable enforcement layer to neural networks for end-to-end training with hard constraint guarantees.
result HardNet retains neural networks' universal approximation capabilities and enables efficient optimization.
The paper explores when to prioritize easy or hard samples in learning tasks.
problem Determining the optimal order of learning easy or hard samples.
method Theoretical analyses and experiments were conducted to propose and validate four priority modes.
result Four priority modes (easy-first, hard-first, medium-first, two-ends-first) can be flexibly applied.
This paper quantifies how hard it is to identify specific data points in machine learning models.
problem Quantifying the difficulty of identifying specific data points in machine learning models.
method Characterizing optimal attacks and privacy defences, deriving impacts of noise and misspecification, and proposing a new covariance attack.
result The Mahalanobis distance explains the hardness of fixed-target membership inference attacks.
Although deep convolutional neural networks achieve state-of-the-art performance across nearly all image classification tasks, their decisions are difficult to interpret. One approach that offers some level of interpretability by design is \textit{hard attention}, which uses only relevant portions of the image. However…
Paper introduces scalable neural architecture for solving NP-hard problems.
problem Solving NP-hard reasoning problems from natural inputs.
method Scalable neural architecture and loss function for discrete Graphical Models.
result Empirically shows efficient learning of NP-hard problems.
Hard Thresholding Pursuit (HTP) is an iterative greedy selection procedure for finding sparse solutions of underdetermined linear systems. This method has been shown to have strong theoretical guarantee and impressive numerical performance. In this paper, we generalize HTP from compressive sensing to a generic problem …
New algorithms minimize regret with multiple best arms in large action spaces.
problem Minimizing regret in multi-armed bandit with multiple best arms.
method Adaptive algorithms that automatically adapt to hardness level, with theoretical regret bounds and lower bounds.
result Proposed algorithms achieve optimal or near-optimal performance, depending on additional knowledge.
Unified framework for hard affine SDP constraints in vRKHSs.
problem Incorporating shape constraints into predictive models for rich function classes.
method Unified convex optimization framework using second-order cone tightening.
result Unified and modular approach for handling multiple shape constraints.
New algorithm resists contamination in high-dimensional regression with optimal performance.
problem Adversarial and measurement errors in high-dimensional data.
method Adversarial Contamination-resistant Iterative Hard Thresholding (AC-IHT) algorithm.
result Achieves minimax near-optimal estimation and signal-adaptive support recovery.
Efficient method for generating adversarial examples with limited query budget.
problem Developing black-box adversarial attacks with limited information.
method Bayesian Optimization in a structured low-dimensional subspace.
result Significantly higher attack success rate with fewer queries.
GACBO optimizes unknown causal graphs with interventions.
problem Optimizing a target variable on an unknown causal graph with interventions.
method Graph Agnostic Causal Bayesian Optimisation (GACBO) seeks to balance exploitation and exploration of causal structures and functions.
result GACBO outperforms baselines in simulated and real-world applications.
Combinatorial auctions are formulated as frustrated lattice gases on sparse random graphs, allowing the determination of the optimal revenue by methods of statistical physics. Transitions between computationally easy and hard regimes are found and interpreted in terms of the geometric structure of the space of solution…
Sharp results link DLN gradient flow to basis pursuit optimization and GHA phase transitions.
problem Understanding implicit regularization in Diagonal Linear Networks.
method Sharp convergence bounds and characterization of ℓ1 minimizers. result Gradient flow of DLNs with tiny initialization approximates minimizers of basis pursuit optimization problem.
Many optimization problems can be cast into the maximum satisfiability (MAX-SAT) form, and many solvers have been developed for tackling such problems. To evaluate a MAX-SAT solver, it is convenient to generate hard MAX-SAT instances with known solutions. Here, we propose a method of generating weighted MAX-2-SAT insta…
Meta-learning strategy improves few-shot classification performance.
problem Few-shot classification with deep neural networks struggles when labeled samples are limited.
method Proposes an easy-to-hard expert meta-training strategy to arrange training tasks based on task hardness.
result Meta-learners achieve better results with the proposed expert training strategy.
Noise makes learning linear thresholds hard, but algorithms can still learn near-optimal thresholds.
problem Learning linear thresholds in noisy data.
method Exploiting natural assumptions on data-generating process.
result Efficient learning of near-optimal linear thresholds is still possible with small data even in the presence of noise.
Active learning optimizes algorithm selection for hard problems.
problem Selecting the best algorithm for hard optimization problems.
method Developed an active learning framework to construct an optimal training set.
result Active learning achieves higher or equal performance with less training data.
Optimal transport is #P-hard when components are independent, even with approximate solutions.
problem Computational complexity of optimal transport with independent marginals.
method Proved #P-hardness and developed a pseudo-polynomial time approximation algorithm.
result Optimal transport is #P-hard even with independent components and approximate solutions.
Boolean logic used for neural network training and inference, with convergence analysis.
problem Discrete optimization in neural networks with Boolean logic.
method Boolean logic backpropagation with convergence analysis.
result First convergence analysis for Boolean logic in neural networks.
New method uses reinforcement learning to improve Simulated Annealing.
problem Optimization problems with unknown cost functions.
method Replaces Metropolis engine with Macau Algorithm.
result Effective heuristic for unknown cost functions.
We study --both in theory and practice-- the use of momentum motions in classic iterative hard thresholding (IHT) methods. By simply modifying plain IHT, we investigate its convergence behavior on convex optimization criteria with non-convex constraints, under standard assumptions. In diverse scenaria, we observe that …
Estimating IPM is as hard as estimating under IPM, both requiring similar optimal rates.
problem Estimating Integral Probability Metrics (IPMs) between probability measures.
method Study of minimax optimal rates for IPM estimation and under IPM estimation based on samples.
result Minimax optimal rates for estimating IPM and estimating under IPM are multiplicatively equivalent.
ARHT algorithm improves sparsity guarantees in convex optimization.
problem Optimizing convex functions with sparsity constraints.
method Adaptively Regularized Hard Thresholding (ARHT) algorithm.
result ARHT achieves sparsity bound of γ=O(κ), matching theoretical limits.
We find optimal learning rate schedules for a random feature model.
problem Choosing optimal learning rates for deep learning models.
method We analyze a powerlaw random feature model trained with SGD, considering optimal schedules as numerical and analytical problems.
result We discover two regimes: easy and hard phases, with different optimal learning rate schedules.
Optimizes hard-to-optimize metrics using adaptive surrogates.
problem Training models with black-box and hard-to-optimize metrics.
method Expresses metric as a function of surrogates, solves optimization problem over relaxed surrogate space.
result Approach performs on par with known methods and adds value when metric form is unknown.
Theoretical comparison of three invariance approaches in deep linear networks.
problem Understanding invariance in deep linear networks.
method Data augmentation, regularization, and hard-wiring approaches.
result Regularization introduces additional critical points, but they remain saddles except for the global optimum.
Optimizes clustering in Gaussian mixtures with varying covariance matrices.
problem Clustering with anisotropic Gaussian mixture models where covariance matrices vary.
method Proposes a computationally feasible hard EM type algorithm.
result Achieves optimal clustering rate with few iterations.
We present a learning-based approach to computing solutions for certain NP-hard problems. Our approach combines deep learning techniques with useful algorithmic elements from classic heuristics. The central component is a graph convolutional network that is trained to estimate the likelihood, for each vertex in a graph…