Two novel network quantization approaches improve deep neural network compression.
problem Efficiently compress deep neural networks for practical usage.
method Single-level network quantization (SLQ) and Multi-level network quantization (MLQ).
result Both SLQ and MLQ achieve impressive results in compressing state-of-the-art neural networks.
Paper explores generalization of AID-based bi-level optimization methods.
problem Uncertainty in generalization properties of AID-based bi-level optimization methods.
method Uniform stability analysis and convergence study of AID-based methods.
result AID-based methods can achieve similar generalization as single-level nonconvex problems.
Proposes a gradient-based bilevel optimization method for efficient hyperparameter tuning.
problem Efficiently tuning hyperparameters in machine learning models.
method Gradient-based bilevel optimization approach.
result The proposed method is multiple times faster than existing techniques.
The paper discusses building ETF risk models using a multilevel classification taxonomy.
problem Building accurate risk models for ETFs.
method First, build a multilevel classification taxonomy for ETFs. Then, use this taxonomy to define risk factors and build risk models.
result The approach can accurately classify and model ETF risks.
Enhances machine learning accuracy with multi-level training data.
problem Improving accuracy of machine learning algorithms for differential equations.
method Combining coarse and fine resolution training data.
result Significant gains in accuracy over single-level algorithms.
Develops a multilevel Monte Carlo framework with dropout for efficient uncertainty quantification.
problem Efficiently quantify uncertainty in complex models using dropout.
method Integrates multilevel Monte Carlo with Monte Carlo dropout, creating coupled estimators to reduce variance.
result Demonstrates significant variance reduction and efficiency gains over single-level Monte Carlo dropout.
Proposes a new clustering algorithm using random forest.
problem Density-based clustering with optimal level determination.
method Best-scored random forest algorithm.
result Guaranteed consistency and fast convergence rates.
Meta-learning balances task-specific modeling and optimization complexity.
problem Balancing accurate task-specific modeling with ease of optimization in meta-learning.
method Theoretical and empirical analysis of trade-off between modeling and optimization in meta-learning.
result Explicit bounds on modeling and optimization errors for non-convex and linear regression problems.
Framework detects covert financial market manipulation using LOB representations.
problem Detecting covert financial market manipulation (spoofing) from complex anomaly patterns in multilevel prices.
method Cascaded contrastive representation learning of LOB data.
result Transformer-based architectures achieve state-of-the-art results in detection performance.
Paper addresses online alignment of large language models under uncertain preference feedback.
problem Online alignment of large language models with misspecified preference feedback.
method Formulates an oracle-robust objective as a worst-case optimization problem for log-linear policies, and develops projected stochastic composite updates.
result Shows that the robust objective admits an exact closed-form decomposition and achieves O ~ ( ε − 2 ) \widetilde{O}(\varepsilon^{-2}) O ( ε − 2 ) oracle complexity. A new algorithm solves bilevel optimization with linear constraints.
problem Solving bilevel optimization problems with coupled linear constraints.
method Penalty and augmented Lagrangian methods reformulate the problem; a single-loop, first-order algorithm proposed.
result Improved convergence rates compared to prior methods.
SAIL-RevKL improves SAIL's convergence by regularizing the objective function.
problem Convergence of self-improving online LLM alignment algorithms.
method Proposed SAIL-RevKL, a regularized objective function to improve optimization landscape.
result Proved SAIL-RevKL satisfies the Polyak-Lojasiewicz (PL) condition with near-linear sample complexity.
A new multilevel framework speeds up ResNet training.
problem Training deep residual networks (ResNets) is time-consuming.
method Formulates ResNets as dynamical systems and uses time-dependent optimal control problems.
result Enhanced training of ResNets with multilevel auxiliary networks achieves significant speedup.
A fast method for fitting deep hierarchical models to large datasets.
problem Fitting a deeply-nested hierarchical model to a large book review dataset.
method Moment-based estimator extension for arbitrarily deep hierarchies.
result Orders of magnitude faster than standard maximum likelihood procedures.
Many problems in sequential decision making and stochastic control often have natural multiscale structure: sub-tasks are assembled together to accomplish complex goals. Systematically inferring and leveraging hierarchical structure, particularly beyond a single level of abstraction, has remained a longstanding challen…
SAIL improves online alignment of large language models with minimal feedback.
problem Offline RLHF methods often lead to sub-optimal performance due to fixed preference datasets.
method SAIL uses bilevel optimization and a single-level first-order method to iteratively refine model alignment.
result SAIL significantly improves alignment performance on open-sourced datasets with minimal computational overhead.
Decision stumps accurately screen variables in nonparametric models.
problem Challenges in theoretical properties of tree-based variable importance measures.
method Derive performance guarantees for variable selection using a single-level CART decision tree (decision stump).
result Decision stumps can perform consistent model selection despite being inaccurate for estimation.
Paper proposes a new method to optimize robot body structure and control policy.
problem Optimizing robot body structure and control policy in a coupled manner.
method Revisits co-design problem as a Stackelberg game, incorporating control adaptation dynamics.
result Stackelberg PPO outperforms standard PPO in stability and performance.
New algorithms improve neural architecture search with faster convergence.
problem Improving efficiency and accuracy of neural architecture search.
method Geometry-aware gradient algorithms to optimize continuous relaxation of discrete search spaces.
result Exceeds state-of-the-art results on CIFAR and ImageNet benchmarks.
Meta Additive Model learns auto-weighting for robust sparse learning.
problem Sparse models' sensitivity to atypical data and noise.
method Bilevel optimization framework with meta data-driven weighting.
result MAM outperforms state-of-the-art models under various data corruptions.
New lower bounds for bilevel optimization with first-order oracles.
problem Complexity of bilevel optimization with first-order oracles.
method Development of hard instances and proof of lower bounds.
result Nontrivial lower bounds for first-order zero-respecting algorithms.
New method solves complex optimization problems faster.
problem Minimizing a convex smooth objective over the optimal solution set of another convex smooth problem.
method Uses a cutting plane approach to approximate the lower-level problem and an accelerated gradient method to update the upper-level objective.
result Shows that the method requires at most O ( max { 1 / ε f , 1 / ε g } ) \mathcal{O}(\max\{1/\sqrt{ε_{f}}, 1/ε_g\}) O ( max { 1/ ε f , 1/ ε g }) iterations to achieve ε f ε_f ε f -suboptimality and ε g ε_g ε g -infeasibility. Develops a new method for efficient stochastic bilevel optimization.
problem Stochastic bilevel optimization problems in machine learning applications.
method Single-Timescale stochAstic BiLevEl optimization (STABLE) method.
result Achieves the same order of sample complexity as stochastic gradient descent for single-level optimization.
This paper shows how procedural level generation improves deep RL models' generalization.
problem Deep RL models overfit to fixed environments, failing to generalize to new levels.
method Procedurally generated levels during training to enhance generalization.
result Procedural level generation enables better generalization to new levels and improved performance with less data.
Paper improves stochastic bilevel optimization methods for highly-smooth problems.
problem Finding ε ε ε -stationary points in stochastic bilevel optimization. method Proposes F 2 {}^2 2 SA- p p p methods using p p p th-order finite differences for hyper-gradient approximation. result Achieves upper complexity bound of i l d e O ( p ε − 4 − p / 2 ) ilde{\mathcal{O}}(p ε^{-4-p/2}) i l d e O ( p ε − 4 − p /2 ) for p p p th-order smooth problems. A method learns to solve multilevel combinatorial problems with two players.
problem Multilevel combinatorial optimization problems with multiple players.
method Value-based multi-agent reinforcement learning in a graph neural network framework.
result Close to optimal solutions on graphs up to 100 nodes, with a significant speedup.
SUSTAIN algorithm tackles stochastic bilevel optimization with near-optimal complexity.
problem Stochastic bilevel optimization problems with specific convexity and smoothness properties.
method SUSTAIN algorithm using single-timescale double-momentum stochastic approximation.
result SUSTAIN achieves near-optimal complexity for finding ε-stationary solutions.
The paper tackles adaptive questioning to classify candidate ability.
problem Designing an optimal sequence of adaptive questions to classify candidate ability.
method Develops lower bounds and geometrical insights for sequential questioning strategies, and arrives at algorithms that match these bounds.
result Asymptotically, any candidate needs to be asked questions at most at two levels, facilitating endogenous exploration.
New algorithms solve complex multi-level optimization problems with improved efficiency.
problem Smooth stochastic multi-level composition optimization problems.
method Two algorithms using moving-average and linearized stochastic estimates.
result Achieved sample complexities of O(1/ε^4) and O(1/ε^6).
Paper tackles efficient SGD methods for constrained bilevel optimization.
problem Stochastic bilevel optimization with equality constraints.
method Alternating implicit projected SGD and its variants.
result Achieves sample complexity matching state-of-the-art for unconstrained problems.
New algorithm tackles stochastic bilevel optimization under relaxed smoothness conditions.
problem Optimal algorithms for stochastic bilevel optimization under relaxed smoothness conditions.
method Introduces a novel fully single-loop and Hessian-inversion-free algorithmic framework for stochastic bilevel optimization.
result Demonstrates state-of-the-art oracle complexity results for multi-objective robust bilevel optimization.
FORE evaluates occupancy ratios without requiring Bellman completeness.
problem Offline reinforcement learning occupancy ratio estimation.
method Fitted occupancy-ratio evaluation (FORE) using adjoint Bellman recursion.
result FORE achieves convergence in KL without Bellman completeness.
New deep fusion methods improve human action recognition using depth and inertial sensor data.
problem Existing multimodal HAR frameworks lack mid-level feature fusion.
method Proposes three deep multilevel multimodal fusion frameworks, transforming depth and inertial sensor data into images and using convolution with Prewitt filter to create modality within modality.
result Supremacy of proposed fusion frameworks over existing methods on three publicly available datasets.
For discrete data, the likelihood P ( x ) P(x) P ( x ) can be rewritten exactly and parametrized into P ( X = x ) = P ( X = x ∣ H = f ( x ) ) P ( H = f ( x ) ) P(X = x) = P(X = x | H = f(x)) P(H = f(x)) P ( X = x ) = P ( X = x ∣ H = f ( x )) P ( H = f ( x )) if P ( X ∣ H ) P(X | H) P ( X ∣ H ) has enough capacity to put no probability mass on any x ′ x' x ′ for which f ( x ′ ) ≠ f ( x ) f(x')\neq f(x) f ( x ′ ) = f ( x ) , where f ( ⋅ ) f(\cdot) f ( ⋅ ) is a deterministic discrete function. The log of the first factor giv…
New methods estimate multivariate shortfall risk more efficiently.
problem Estimating multivariate shortfall risk is computationally challenging.
method Combines Fourier inversion and RQMC sampling in frequency domain.
result Fourier RQMC methods outperform existing benchmarks.
RARTS improves neural architecture search efficiency and accuracy.
problem Efficiency and accuracy issues in first-order DARTS.
method Formulates a relaxed architecture search (RARTS) method that uses whole dataset splitting.
result RARTS outperforms DARTS and its variants in accuracy and efficiency.