arXiv research

The paper provides convergence guarantees for multicalibration gradient boosting.

Paper proposes a new method for training nonconvex models.

Momentum Stochastic Gradient Descent (MSGD) algorithm has been widely applied to many nonconvex optimization problems in machine learning, e.g., training deep neural networks, variational Bayesian inference, and etc. Despite its empirical success, there is still a lack of theoretical understanding of convergence proper…

Neural networks extrapolate poorly in simple tasks but succeed in complex ones.

This work improves SGMs' convergence guarantees for semiconvex distributions with discontinuous gradients.

Student-teacher learning improves generalization with noisy inputs.

Machine learning predicts TV show success based on factors like characters and direction.

New method uses robust estimators for Newton's method in empirical risk minimization.

Streaming variational Bayes (SVB) is successful in learning LDA models in an online manner. However previous attempts toward developing online Monte-Carlo methods for LDA have little success, often by having much worse perplexity than their batch counterparts. We present a streaming Gibbs sampling (SGS) method, an onli…

A new method sorts models to find the best one with minimal risk.

Proposes a cross entropy loss for better ranking algorithms.

Predicting startup success using Crunchbase data and deep learning.

AdaBoost's success explained through noise influence measure.

Graph neural networks improve solving linear optimization problems.

Random forests use randomness to improve model performance in noisy data.

Belief Propagation algorithms are instruments used broadly to solve graphical model optimization and statistical inference problems. In the general case of a loopy Graphical Model, Belief Propagation is a heuristic which is quite successful in practice, even though its empirical success, typically, lacks theoretical gu…

The success of modern Artificial Intelligence (AI) technologies depends critically on the ability to learn non-linear functional dependencies from large, high dimensional data sets. Despite recent high-profile successes, empirical evidence indicates that the high predictive performance is often paired with low robustne…

Random Reshuffling outperforms Stochastic Gradient Descent in smooth convex optimization.

ReLU neural networks define piecewise linear functions of their inputs. However, initializing and training a neural network is very different from fitting a linear spline. In this paper, we expand empirically upon previous theoretical work to demonstrate features of trained neural networks. Standard network initializat…

New framework explains why over-parameterized neural networks work well.

Improved sample complexity for diffusion models without needing empirical risk minimizers.

Distributional reinforcement learning (distributional RL) has seen empirical success in complex Markov Decision Processes (MDPs) in the setting of nonlinear function approximation. However, there are many different ways in which one can leverage the distributional approach to reinforcement learning. In this paper, we p…

The study provides a generalization bound for a family of implicit networks.

Despite the huge empirical success of deep learning, theoretical understanding of neural networks learning process is still lacking. This is the reason, why some of its features seem "mysterious". We emphasize two mysteries of deep learning: generalization mystery, and optimization mystery. In this essay we review and …

We analyze the probability density function (PDF) of waiting times between financial loss exceedances. The empirical PDFs are fitted with the self-excited Hawkes conditional Poisson process with a long power law memory kernel. The Hawkes process is the simplest extension of the Poisson process that takes into account h…

Multiplicative random cascade model naturally reproduces the intermittency or multifractality, which is frequently shown among hierarchical complex systems such as turbulence and financial markets. As described herein, we investigate the validity of a multiplicative hierarchical random cascade model through an empirica…

Recently, a unified model for image-to-image translation tasks within adversarial learning framework has aroused widespread research interests in computer vision practitioners. Their reported empirical success however lacks solid theoretical interpretations for its inherent mechanism. In this paper, we reformulate thei…

Indirect competition emerged from the complex organization of human societies, and knowledge of the existing network topology may aid in developing effective strategies for success. Here, we propose an agent-based model of competition with systems co-existing in a `small-world' social network. We show that within the r…

RLHF performs well despite violating social choice theory axioms.

In high dimensional settings, density estimation algorithms rely crucially on their inductive bias. Despite recent empirical success, the inductive bias of deep generative models is not well understood. In this paper we propose a framework to systematically investigate bias and generalization in deep generative models …

When machine predictors can achieve higher performance than the human decision-makers they support, improving the performance of human decision-makers is often conflated with improving machine accuracy. Here we propose a framework to directly support human decision-making, in which the role of machines is to reframe pr…

Paper bridges $f$-GANs and WGANs for better image generation.

SSL framework identifies non-linear systems without labeled data.

New convergence rates for shuffling gradient methods without strong convexity.

This work compares human feedback methods for reward learning in bandits.

Wasserstein GANs fail to approximate Wasserstein distance, leading to their success.

New bound limits generalization gap for large models, independent of model complexity.

Proposes a method to select features for deep learning in noisy, high-dimensional data.

The paper explores theoretical insights into WGANs for better understanding and stability.

The convergence rate and final performance of common deep learning models have significantly benefited from heuristics such as learning rate schedules, knowledge distillation, skip connections, and normalization layers. In the absence of theoretical underpinnings, controlled experiments aimed at explaining these strate…

Survey of data augmentation techniques for time series classification with neural networks.

A novel version of the Continuous-Time Random Walk (CTRW) model with memory is developed. This memory means the dependence between arbitrary number of successive jumps of the process, while waiting times between jumps are considered as i.i.d. random variables. The dependence was found by analysis of empirical histogram…

Paper analyzes convergence of DDPM for general distributions.

Improved adaptive algorithms for identifying the best arm in MABs with fixed budget.

New algorithm finds best feasible arm in grouped bandits.

This paper theoretically explains and validates a deep neural network approach to IV estimation.

Deep neural networks outperform traditional methods in high-dimensional classification.

Despite the great empirical success of deep reinforcement learning, its theoretical foundation is less well understood. In this work, we make the first attempt to theoretically understand the deep Q-network (DQN) algorithm (Mnih et al., 2015) from both algorithmic and statistical perspectives. In specific, we focus on …