Wide deep neural networks are easy to optimize without constraints.
problem Optimizing wide deep neural networks.
method Analysis of optimization landscapes and empirical-risk minimization.
result Wide neural networks have no confined points, making optimization easier.
We solve the optimization of two-layer ReLU networks using convex math.
problem Optimizing two-layer ReLU neural networks.
method Exact characterization of optimal solutions via convex optimization.
result We prove that all globally optimal solutions can be found via convex optimization.
Optimizes control interventions in real-world networks using deep-learning and network science.
problem Optimizing control over socioeconomic networks subject to constraints.
method Integrates optimization tools from deep-learning with network science.
result Characterizes vulnerability of corporate networks to takeovers.
GT-DDP optimizer trains residual networks using game theory.
problem Applying DDP to residual networks.
method Game-theoretic DDP optimizer for residual networks.
result Improved training convergence and variance reduction.
Optimizes function networks by selectively evaluating parts of the network, reducing costs.
problem Optimizing expensive function networks where full evaluations are costly.
method Knowledge gradient acquisition function for cost-aware partial evaluations.
result Outperforms existing BOFN methods and benchmarks across various problems.
LiPopt uses polynomial optimization to estimate neural network Lipschitz constants efficiently.
problem Estimating the Lipschitz constant of neural networks efficiently.
method Sparse polynomial optimization, leveraging network connectivity to reduce complexity.
result Superior estimates of the ℓ∞-Lipschitz constant compared to existing methods. This paper presents a widely applicable approach to solving (multi-marginal, martingale) optimal transport and related problems via neural networks. The core idea is to penalize the optimization problem in its dual formulation and reduce it to a finite dimensional one which corresponds to optimizing a neural network wi…
Paper tackles optimal network compression for financial systems.
problem Optimal network compression for financial systems under shocks.
method Formulated as an NP-hard problem, studied systemic risk measures, and analyzed specific networks.
result Systemic fragility results no longer hold generally under shocks and heterogeneous networks.
Optimal rates for shallow ReLU networks in nonparametric regression.
problem Approximating smooth and non-smooth functions with shallow ReLU networks.
method Analysis of shallow ReLUk neural networks, using variation norms and deep learning theory. result Optimal approximation rates for shallow ReLU networks in nonparametric regression.
Frank-Wolfe optimization applied to a small deep network shows slower convergence compared to gradient descent.
problem Training deep neural networks is challenging due to many parameters and optimization difficulties.
method Frank-Wolfe optimization method applied to a deep network.
result Frank-Wolfe optimization converges slowly and is unstable in a stochastic setting.
Deep neural networks (DNNs) are powerful machine learning models and have succeeded in various artificial intelligence tasks. Although various architectures and modules for the DNNs have been proposed, selecting and designing the appropriate network structure for a target problem is a challenging task. In this paper, w…
Joslim optimizes both width and weight configurations for slimmable neural networks, improving model efficiency.
problem Optimizing both width and weight configurations for slimmable neural networks to improve efficiency.
method Proposes a general framework for joint optimization of width configurations and weights, and introduces Joslim algorithm.
result Improves model efficiency by up to 1.7% in top-1 accuracy on the ImageNet dataset.
Researchers improve transformer networks' optimization and understanding.
problem Improving the understanding and optimization of transformer networks.
method Introducing a convex alternative to the self-attention mechanism and reformulating the training problem as a convex optimization problem.
result Revealed an implicit regularization mechanism that promotes sparsity across tokens.
Optimizes graph neural networks using natural gradient descent.
problem Improving efficiency and performance of graph neural networks.
method Employing natural gradient descent to optimize graph neural networks.
result Natural gradient optimization leads to superior performance compared to existing methods.
Optimization algorithms help overparameterized neural networks achieve high performance.
problem Understanding the convergence of optimization algorithms on overparameterized neural networks.
method Analyzing a broad class of optimization algorithms using dynamical systems and finite over-parameterized neural networks with ReLU activation.
result The Heavy Ball method converges to global minimum at a linear rate, while NAG converges sublinearly.
GAIL with neural networks converges to global optima and has a known rate.
problem Uncertainty about GAIL with neural networks achieving global optimality.
method Gradient-based alternating updates algorithm.
result Established sublinear convergence to globally optimal solution.
A novel neural network approach for optimization problems.
problem Constrained optimization problems.
method Neural Optimization Machine (NOM) using a specially designed NN architecture and training procedure.
result Solves optimization problems efficiently, especially in high-dimensional spaces.
The robustness and integrity of IP networks require efficient tools for traffic monitoring and analysis, which scale well with traffic volume and network size. We address the problem of optimal large-scale flow monitoring of computer networks under resource constraints. We propose a stochastic optimization framework wh…
A new distributed method for convex optimization over networks with fast convergence.
problem Large-scale convex optimization over networks with limited communication.
method Distributed cubic-regularized Newton method.
result Convergence rate of O(k−3) for convex functions with Lipschitz gradient and Hessian. Gradient descent methods for deep ReLU networks achieve optimal generalization rates.
problem Generalization of gradient descent methods for deep neural networks
method Establishing minimax-optimal rates for GD and SGD with deep ReLU networks
result Gradient descent methods for deep ReLU networks achieve optimal generalization rates
Ensembles of neural networks learn better by sharing information.
problem Improving performance of neural networks through collective learning.
method Modeling neural networks as socially interacting agents aiming to maximize their own performance and functional relations to others.
result Optimal collective performance emerges from local interactions between networks, leading to specialization and higher confidence.
Paper optimizes clustering for multi-layer networks and discrete mixtures.
problem Optimizing clustering in multi-layer networks and discrete mixtures.
method Two-stage method: tensor-based initialization and likelihood-based refinement.
result Achieves minimax optimal error rate for multi-layer networks and discrete mixtures.
Improved hypernetwork for efficient neural network hyperparameter tuning.
problem Efficiently optimizing hyperparameters in neural networks.
method Proposed Δ-STN architecture focusing on accurate best-response Jacobian approximation. result Significantly improved hyperparameter tuning accuracy and stability.
Optimal stock price prediction model using recurrent neural networks with RMSprop optimizer.
problem Stock price prediction using neural networks.
method Comparison of fully connected, convolutional, and recurrent architectures; inclusion of three optimization techniques.
result Single layer recurrent neural network with RMSprop optimizer produces optimal results with validation and test MAE of 0.0150 and 0.0148 respectively.
This paper proposes a gossip-based algorithm for distributed bilevel optimization over networks.
problem Distributed bilevel optimization over networks with limited communication.
method Gossip-based distributed bilevel learning algorithm.
result Achieves optimal sample complexities for general and strongly convex objectives.
Deep neural networks can solve optimal stopping problems without dimensionality issues.
problem Optimal stopping problems in high-dimensional state spaces.
method Established a general framework for deep ReLU neural networks to approximate value functions and continuation values.
result Deep neural networks can approximate value functions and continuation values with error at most ε of size κd^q ε^(-r).
Deep neural networks are traditionally trained using human-designed stochastic optimization algorithms, such as SGD and Adam. Recently, the approach of learning to optimize network parameters has emerged as a promising research topic. However, these learned black-box optimizers sometimes do not fully utilize the experi…
The goal of decentralized optimization over a network is to optimize a global objective formed by a sum of local (possibly nonsmooth) convex functions using only local computation and communication. It arises in various application domains, including distributed tracking and localization, multi-agent co-ordination, est…
We consider the problem of downlink power control in wireless networks, consisting of multiple transmitter-receiver pairs communicating with each other over a single shared wireless medium. To mitigate the interference among concurrent transmissions, we leverage the network topology to create a graph neural network arc…
Study on deep neural networks using concentration inequalities and optimal stopping.
problem Understanding the performance and structure of stochastic deep neural networks.
method Introduced concentration inequalities for SDNN outputs and an EC classifier. Determined the optimal number of layers via an optimal stopping procedure.
result Optimal number of layers for SDNNs determined via an optimal stopping procedure.
Unified framework reveals regularization mechanism in deep ReLU networks via convex optimization.
problem Understanding the success of deep neural networks.
method Developed a unified framework using convex optimization to reveal regularization mechanisms.
result ReLU networks can be globally optimized via convex programs, enforcing sparsity.
Path regularization reveals convex optimization in deep ReLU networks.
problem Understanding the optimization landscape of deep neural networks.
method Introducing path regularization to make the training problem convex and sparsity-inducing.
result Path regularized parallel ReLU networks are a parsimonious convex model in high dimensions.
Optimal probing framework for scalable network monitoring.
problem Efficiently monitor growing cloud networks with limited budgets.
method A- and E-optimal experimental designs, Frank-Wolfe algorithm approximations.
result Significant reduction in probing budget with low estimation errors.
Deep neural networks solve optimal risk sharing problems.
problem Optimally sharing financial positions among agents with different risk measures.
method Neural network-based framework to compute inf-convolution and optimal allocations.
result Convergence of neural network approximations to theoretical values.
Bayesian optimization on networks using Gaussian process models.
problem Optimizing expensive black-box functions on network structures.
method Developed Bayesian optimization algorithms with Gaussian process surrogates tailored to network geometry.
result Established regret bounds for smooth objective functions and analyzed practical cases.
The paper proposes using network science to improve portfolio optimization by reducing noise in covariance estimation.
problem Noise in covariance estimation leads to suboptimal portfolio performance.
method The paper introduces SR-IFN, a network-based method to filter out noise from empirical covariance, enhancing portfolio optimization.
result The SR-IFN network improves portfolio performance by selecting peripheral, diversified assets and inversely weighting them based on centrality.
We propose a new scalable method to optimize the architecture of an artificial neural network. The proposed algorithm, called Greedy Search for Neural Network Architecture, aims to determine a neural network with minimal number of layers that is at least as performant as neural networks of the same structure identified…
A new Kolmogorov-Arnold network improves function approximation and optimization.
problem Approximating potentially irregular functions in high dimensions.
method Proposes a new Kolmogorov-Arnold network (KAN) and provides error bounds and universal approximation theorems.
result Outperforms multilayer perceptrons in accuracy and convergence speed for irregular functions.
We present an approach to obtain convergence guarantees of optimization algorithms for deep networks based on elementary arguments and computations. The convergence analysis revolves around the analytical and computational structures of optimization oracles central to the implementation of deep networks in machine lear…
HyCNNs improve convex function learning and optimal transport.
problem Learning and optimizing convex functions efficiently.
method Combining Maxout networks and ICNNs to create a new neural architecture.
result HyCNNs require fewer parameters and outperform existing methods in convex tasks.
Deep neural networks solve stochastic control problems with delay.
problem Challenges in stochastic control problems with delay due to path-dependence and high dimensions.
method Employing recurrent neural networks (RNNs) to parameterize policies and optimize objectives.
result RNNs, especially LSTMs, efficiently capture path-dependence and outperform feedforward networks in training and performance.
Optimizes wireless network resource management with state-augmented policies.
problem Optimizing network-wide utility with user performance constraints.
method State-augmented parameterization of RRM policy, using dual variables.
result Superior trade-off between mean, minimum, and 5th percentile rates.
Neural network predicts optimal pension investments based on preferences.
problem Optimal pension investment problem with varying preferences.
method Used a neural network to identify optimal solutions to a family of investment problems.
result Validated network accuracy using classical numerical methods.
Optimizing a neural network's performance is a tedious and time taking process, this iterative process does not have any defined solution which can work for all the problems. Optimization can be roughly categorized into - Architecture and Hyperparameter optimization. Many algorithms have been devised to address this pr…
Bayes-optimal learning of deep random networks with Gaussian weights is studied.
problem Learning a target function corresponding to a deep, extensive-width, non-linear neural network with random Gaussian weights.
method Closed-form expressions for Bayes-optimal test error, ridge regression, kernel and random features regression are computed.
result Optimally regularized ridge regression and kernel regression achieve Bayes-optimal performances, while logistic loss yields a near-optimal test error for classification.
Novel approach analyzes ReLU networks' training dynamics and proposes GmP for improved optimization.
problem Stochastic optimization instability in ReLU networks impedes convergence and generalization.
method Characteristic activation boundaries analysis and Geometric Parameterization (GmP) technique.
result GmP resolves instability, leading to better optimization, convergence, and generalization.
Graph neural networks improve solving linear optimization problems.
problem Improving the efficiency of solving linear optimization problems.
method Using graph neural networks to simulate standard interior-point methods for linear optimization problems.
result Graph neural networks can solve linear optimization problems close to optimality, often outperforming conventional solvers.
Muon optimizer outperforms GD in neural networks.
problem Optimizing matrix-structured parameters in neural networks.
method Muon optimizer specifically designed for matrix parameters, analyzing convergence rate and low-rank Hessian structure.
result Muon can outperform Gradient Descent due to its ability to leverage the low-rank structure of Hessian matrices.