Hypersolvers enable fast continuous-depth models for practical applications.
problem Infinite-depth models like Neural ODEs are computationally infeasible for large problems.
method Introducing hypersolvers, neural networks that solve ODEs efficiently with theoretical guarantees.
result Hypersolvers achieve comparable inference time to traditional discrete networks, making continuous-depth models practical.
We introduce a new family of deep neural network models. Instead of specifying a discrete sequence of hidden layers, we parameterize the derivative of the hidden state using a neural network. The output of the network is computed using a black-box differential equation solver. These continuous-depth models have constan…
GoTube verifies neural networks over time, scaling to large horizons.
problem Verifying the robustness of time-continuous neural networks.
method Solves Go problems to construct a conservative execution set.
result Substantially outperforms existing tools in size, speed, and scalability.
Continuous-depth Evoformer reduces protein folding prediction time and resource usage.
problem Efficient protein structure prediction with reduced computational costs.
method Continuous-depth formulation of Evoformer using Neural Ordinary Differential Equations (Neural ODEs).
result The continuous-time Evoformer achieves constant memory cost and improved efficiency.
Neural GDEs improve graph prediction by blending discrete structures and differential equations.
problem Dynamic graph prediction challenges in irregularly sampled data.
method Continuous-depth graph neural networks (GNNs) with Neural GDEs.
result Neural GDEs enhance performance across various applications.
We develop a scalable method for Bayesian neural networks with stochastic differential equations.
problem Uncertainty quantification in deep neural networks.
method Gradient-based stochastic variational inference in continuous-depth Bayesian neural networks.
result Gradient estimator with zero variance as the approximation improves.
Continuous deep learning architectures have recently re-emerged as Neural Ordinary Differential Equations (Neural ODEs). This infinite-depth approach theoretically bridges the gap between deep learning and dynamical systems, offering a novel perspective. However, deciphering the inner working of these models is still a…
We introduce the framework of continuous--depth graph neural networks (GNNs). Graph neural ordinary differential equations (GDEs) are formalized as the counterpart to GNNs where the input-output relationship is determined by a continuum of GNN layers, blending discrete topological structures and differential equations.…
Revisits Gaussian process model with spherical harmonics for scalable deep learning.
problem Scaling Gaussian process models to large input dimensions with high frequency learning.
method Introduces new kernels related to deep models, variational learning of spherical harmonic phases, and sparseness in eigenbasis.
result Enables scaling to larger input dimensions and learning of high frequency variations.
Generalization bounds derived for neural ODEs and deep residual networks.
problem Understanding the generalization capability of neural ODEs and deep residual networks.
method Lipschitz-based argument and analogy with deep residual networks.
result A generalization bound involving the magnitude of weight matrix differences.
Improved SDE-BNN model reduces NFEs and accelerates convergence.
problem High computational cost and convergence instability in SDE-BNNs.
method Nesterov's Accelerated Gradient (NAG) method integrated into SDE-BNN framework.
result Significantly reduced number of function evaluations (NFEs) and improved predictive accuracy.
KuramotoGNN uses Kuramoto model to prevent over-smoothing in graph neural networks.
problem Over-smoothing in graph neural networks where node features become indistinguishable.
method Integrates Kuramoto model to prevent phase synchronization and instead achieve frequency synchronization.
result KuramotoGNN reduces over-smoothing on various graph deep learning tasks.
Neural ODEs simplified using Chen-Fliess series for Rademacher complexity analysis.
problem Analyzing the complexity of neural ODE models.
method Using Chen-Fliess series to frame neural ODEs as infinite-width nets, where weights are signature of control input and features are Lie derivatives.
result Derived compact expressions for the Rademacher complexity of ODE models.
Deep residual networks implicitly converge to neural ODEs.
problem Link between discrete and continuous deep learning models.
method Establishing implicit regularization for residual networks towards neural ODEs.
result Deep residual networks initialized as discretizations of neural ODEs converge to such ODEs during training.
Improved neural-ODE for faster convergence and stability.
problem Stability, consistency, and convergence issues in neural-ODE solvers.
method Proposed a first-order Nesterov's accelerated gradient (NAG) based ODE-solver.
result Efficacy demonstrated in three tasks: supervised classification, density estimation, and time-series modelling.
Study Transformer layers under cross-entropy training using mean field control.
problem Understanding the behavior of Transformer layers in cross-entropy training.
method Continuous-depth mean field control analysis, treating depth as time and layer parameters as controls.
result Derivation of a Pontryagin condition for the limiting population problem, involving the softmax residual.
Develops EFT for ResNets, revealing limitations of kernel-only approach.
problem Limitations of kernel-only approach in deep neural networks.
method Collective kernel EFT for pre-activation ResNets based on G-only closure hierarchy. result Numerical findings show V4 equation residual accumulates to an O(1) error. Proposes a method to model uncertainty in neural ordinary differential equations.
problem Lack of uncertainty modeling and robustness in neural ordinary differential equations.
method Introduces a novel approach to model uncertainty by considering a distribution over the end-time of the ODE solver.
result Demonstrates the effectiveness of the proposed approaches in modelling uncertainty and robustness through experiments.
Deep neural networks can approximate complex functions through repeated compositions of a fixed-size ReLU network.
problem Understanding the expressive power of deep neural networks through function compositions.
method Demonstrated the surprising expressive power of repeated compositions of a single fixed-size ReLU network.
result Repeated compositions of a single fixed-size ReLU network can approximate 1-Lipschitz continuous functions on [0,1]d with an error O(r−1/d). The study tests a functional-form restriction on risk exposure dynamics using margin debt data.
problem Understanding risk exposure dynamics under capital constraints and slack.
method Testing a regime-conditional functional-form restriction on aggregate risk-exposure dynamics implied by VaR-constrained intermediary models.
result The contraction and growth of exposures under capital constraints and slack are observed and tested.
The paper introduces BCART models for aggregate claim amount, improving frequency-severity and joint modeling.
problem Modeling aggregate claim amount with frequency-severity and joint dependencies.
method Developed three types of BCART models: frequency-severity, sequential, and joint models. Used various distributions for claim severity data.
result Weibull distribution outperforms gamma and lognormal for right-skewed, heavy-tailed claim severity data.
The paper uses model-based trees to create interpretable surrogate models for complex machine learning models.
problem Interpreting complex machine learning models.
method Using model-based trees to partition feature space and create interpretable models.
result Model-based trees generate optimal surrogate models that balance interpretability and performance.
Gauge Flow Models use a learnable Gauge Field in Generative Flow Models.
problem Improving generative model performance.
method Integrates a learnable Gauge Field into Flow ODEs.
result Gauge Flow Models outperform traditional Flow Models in Flow Matching experiments.
The study examines how model predictions hold up under model extensions.
problem Model predictions may not be robust under model extensions, limiting their applicability.
method The study uses causal ordering to assess robustness of qualitative model predictions and characterizes model extensions that preserve predictions.
result Conditions and techniques are provided to assess robustness of model predictions under model extensions.
Revises Bayesian model averaging for foundation models.
problem Ensemble pre-trained and lightly-finetuned foundation models for improved classification performance.
method Introduces trainable linear classifiers and computationally cheaper model averaging scheme (OMA).
result Ensembled models can better predict on various datasets.
Paper introduces symmetric divergence link models for probability distributions.
problem Symmetric divergence measures for probability distributions.
method Two general classes of link models: one for survival functions and another for cumulative probability distribution functions.
result Advantages of symmetric divergence measures over asymmetric measures for model averaging and feature assessment.
New method to handle credit portfolio model uncertainties.
problem Model risk in credit portfolio models.
method Demonstrates comprehensive yet easy-to-implement approach to uncertainty in model parameters.
result Comprehensive method to deal with model uncertainties.
The paper tests stock return models and uses LSTM to predict stock returns.
problem Validating stock return models and predicting stock returns.
method Used Fama-French three-factor, four-factor, and five-factor models; also used LSTM model.
result Fama-French five-factor model shows better validity for stock returns.
Researchers review challenges in interpreting additive models, especially neural additive models.
problem Challenges in interpreting additive models, particularly neural additive models.
method Review of generalized additive models and discussion of nonidentifiability.
result Challenges in claiming interpretability or suitability for safety-critical applications of additive models.
Novel hybrid modeling combines ML and physics for real-time diagnosis.
problem Real-time diagnosis of complex systems.
method Combines machine learning and physics-based models to create reduced-order models.
result Generated models are two orders of magnitude simpler, improving efficiency.
CRS model improves ranking data modeling with theoretical guarantees.
problem Lack of rich, multimodal models for ranking data.
method Contextual Repeated Selection (CRS) model for multimodal ranking data.
result CRS model significantly outperforms existing methods in various ranking contexts.
Sigma models linked to Gross-Neveu models via quiver varieties.
problem Understanding the relationship between sigma models and Gross-Neveu models.
method Exploring the mathematical correspondence between sigma models and Gross-Neveu models, including their geometric and trigonometric/elliptic deformations.
result Sigma models are mathematically equivalent to Gross-Neveu models under certain conditions.
Interpretable machine learning has become a strong competitor for traditional black-box models. However, the possible loss of the predictive performance for gaining interpretability is often inevitable, putting practitioners in a dilemma of choosing between high accuracy (black-box models) and interpretability (interpr…
Simple models are preferred over complex models, but over-simplistic models could lead to erroneous interpretations. The classical approach is to start with a simple model, whose shortcomings are assessed in residual-based model diagnostics. Eventually, one increases the complexity of this initial overly simple model a…
Matryoshka hides secret models in a carrier model, achieving high capacity and robustness.
problem Stealing functionality of private ML data by hiding models in a carrier model.
method Parameter sharing approach exploiting the learning capacity of the carrier model.
result Hides a 26x larger secret model or 8 secret models in the carrier model.
Seq2Seq models speed up epidemic model predictions.
problem Complex epidemic models are computationally expensive.
method Used deep seq2seq models as surrogates for complex models.
result Surrogates predict scenarios up to several thousand times faster.
This work develops scalable model selection methods with fast update and selection.
problem Efficient model selection for large pools of candidate models.
method Isolated model embedding, which supports asymptotically fast update and selection.
result Standardized Embedder achieves competitive model selection performances.
Paper proposes BMPO to optimize policies using bidirectional models.
problem Model-based reinforcement learning's reliance on forward model accuracy.
method Develops BMPO using both forward and backward models for policy optimization.
result BMPO outperforms state-of-the-art methods in sample efficiency and asymptotic performance.
Copulas outperform marginal models in multivariate risk forecasting, reducing model risk by narrowing down the set of models.
problem Model risk in multivariate risk forecasting, especially during crises.
method Comprehensive empirical study comparing Copula-GARCH models with fixed marginals, copulas, or neither.
result Model risk is almost entirely due to copula choice, not marginal models.
BayesBlend blends multiple models' predictions for better insurance loss predictions.
problem Improving insurance loss predictions by combining multiple models.
method Pseudo-Bayesian model averaging, stacking, and hierarchical stacking.
result BayesBlend provides a user-friendly way to blend model predictions and estimate weights.
The paper identifies when larger models improve predictions and proposes a switcher model.
problem Understanding when larger models benefit from added complexity.
method Numerical studies on T5 architecture to analyze predictive uncertainty and model performance.
result Large models improve on examples where small models are uncertain, but not on certain examples.
Improved diffusion model generation speed with speculative sampling.
problem Generating samples from computationally expensive diffusion models.
method Extending speculative sampling to diffusion models, using fast draft models for candidate token generation.
result Significant speedup in generation, halving the number of function evaluations.
We propose a generalization of neural network sequence models. Instead of predicting one symbol at a time, our multi-scale model makes predictions over multiple, potentially overlapping multi-symbol tokens. A variation of the byte-pair encoding (BPE) compression algorithm is used to learn the dictionary of tokens that …
The paper extends statistical inference methods for black-box generative models.
problem Understanding and validating black-box generative models without access to their internal data.
method Develops model-level statistical inference tasks using generative model representations.
result Model-level representations are effective for multiple inference tasks.
PMM uses Bayesian inference to generate data from noisy approximations.
problem Creating flexible generative models for various data types.
method Bayesian inference and conjugate pairs of distributions.
result PMM achieves performance competitive with existing generative models.
Unified model improves sampling speed and quality.
problem Difficult to balance sampling speed and quality.
method Multistep Consistency Models combining consistency and diffusion models.
result Improved sampling quality with reduced steps.
In science and especially in economics, agent-based modeling has become a widely used modeling approach. These models are often formulated as a large system of difference equations. In this study, we discuss two aspects, numerical modeling and the probabilistic description for two agent-based computational economic mar…
Driven by an increasing need for model interpretability, interpretable models have become strong competitors for black-box models in many real applications. In this paper, we propose a novel type of model where interpretable models compete and collaborate with black-box models. We present the Model-Agnostic Linear Comp…