Overparameterisation can limit the benefits of curriculum learning in neural networks.
problem The ineffectiveness of curriculum learning in deep learning applications.
method Analytical study connecting curriculum learning and overparameterisation in an online learning setting for a 2-layer network in the XOR-like Gaussian Mixture problem.
result High degree of overparameterisation can limit the benefit from curricula.
Depth uncertainty networks don't improve with bias correction, contrary to expectations.
problem Improving performance in active learning with overparameterised models like NNs.
method Depth uncertainty networks, compared to underparameterised models, show no improvement in performance with bias correction.
result Depth uncertainty networks do not improve with bias correction, unlike underparameterised models.
Study reveals sharp characterisation of local minima in neural network loss landscapes.
problem Characterizing local minima in high-dimensional two-layer ReLU neural networks.
method Exact low-dimensional representation of local minima using summary statistics and link with one-pass SGD dynamics.
result Local minima in overparameterized neural networks form discrete families with varying stability and reachability.
New stability bounds for GD in overparameterised shallow nets without NTK assumptions.
problem Generalisation and excess risk bounds for shallow neural networks.
method Oracle inequalities and stability analysis of GD without kernelisation.
result Oracle type bounds reveal GD's generalisation is controlled by an interpolating network with shortest GD path.
SGD-trained deep nets often generalize well due to a strong inductive bias towards low-error, low-complexity functions.
problem Understanding why overparameterized deep nets generalize well despite fitting training data perfectly.
method Empirical investigation of PSGD(f∣S) and PB(f∣S) for various architectures and datasets. result The probability of SGD-converging on a function consistent with training data correlates well with the Bayesian posterior probability of expressing that function.
New research shows non-bottlenecked autoencoders can outperform bottlenecked ones for anomaly detection.
problem The necessity of a bottleneck in autoencoders for anomaly detection.
method Investigated two ways to remove bottlenecks: overparameterising the latent layer and introducing skip connections. Carried out extensive experiments on various AE types and datasets.
result Non-bottlenecked autoencoders can outperform bottlenecked ones, improving anomaly detection performance.
New bounds link flat minima to good generalisation in overparameterized models.
problem Understanding the relationship between flat minima and generalisation in overparameterized machine learning models.
method Combining PAC-Bayes, Poincaré, and Log-Sobolev inequalities to derive generalisation bounds involving gradient terms.
result Flat minima positively influence generalisation performance, highlighting the benefits of the optimisation phase.
Early alignment in neural networks leads to sparse representations but hinders convergence.
problem The implicit bias of gradient descent during early training phases.
method Quantitative description of early alignment phase in small initialisation, one hidden layer networks.
result Early alignment induces a sparse representation but also hinders convergence to global minima.
Analyzes SGD dynamics in two-layer networks, bridging different regimes.
problem Understanding SGD dynamics in high-dimensional and mean-field settings.
method Rigorous analysis via deterministic low-dimensional description of sufficient statistics.
result Infinite-width dynamics remains close to a low-dimensional subspace.
New kernels from ELU and GELU networks reveal non-trivial fixed points.
problem Understanding fixed-point dynamics in deep neural networks with ELU and GELU activations.
method Deriving covariance functions and analyzing fixed-point dynamics of ELU and GELU networks.
result ELU and GELU networks exhibit non-trivial fixed-point dynamics, explaining implicit regularization in overparameterized models.
A new geometric concept, the dead direction, bridges singular learning theory and information geometry.
problem The gap between singular learning theory and information geometry.
method Introducing the dead direction, a unit vector along degenerating Fisher metric, and showing its KL order can be recovered.
result The KL order of the dead direction can be recovered as the decay rate of the directional Fisher curvature, providing a handle on singular geometry.
The paper introduces BCART models for aggregate claim amount, improving frequency-severity and joint modeling.
problem Modeling aggregate claim amount with frequency-severity and joint dependencies.
method Developed three types of BCART models: frequency-severity, sequential, and joint models. Used various distributions for claim severity data.
result Weibull distribution outperforms gamma and lognormal for right-skewed, heavy-tailed claim severity data.
The paper uses model-based trees to create interpretable surrogate models for complex machine learning models.
problem Interpreting complex machine learning models.
method Using model-based trees to partition feature space and create interpretable models.
result Model-based trees generate optimal surrogate models that balance interpretability and performance.
Gauge Flow Models use a learnable Gauge Field in Generative Flow Models.
problem Improving generative model performance.
method Integrates a learnable Gauge Field into Flow ODEs.
result Gauge Flow Models outperform traditional Flow Models in Flow Matching experiments.
The study examines how model predictions hold up under model extensions.
problem Model predictions may not be robust under model extensions, limiting their applicability.
method The study uses causal ordering to assess robustness of qualitative model predictions and characterizes model extensions that preserve predictions.
result Conditions and techniques are provided to assess robustness of model predictions under model extensions.
Revises Bayesian model averaging for foundation models.
problem Ensemble pre-trained and lightly-finetuned foundation models for improved classification performance.
method Introduces trainable linear classifiers and computationally cheaper model averaging scheme (OMA).
result Ensembled models can better predict on various datasets.
Paper introduces symmetric divergence link models for probability distributions.
problem Symmetric divergence measures for probability distributions.
method Two general classes of link models: one for survival functions and another for cumulative probability distribution functions.
result Advantages of symmetric divergence measures over asymmetric measures for model averaging and feature assessment.
New method to handle credit portfolio model uncertainties.
problem Model risk in credit portfolio models.
method Demonstrates comprehensive yet easy-to-implement approach to uncertainty in model parameters.
result Comprehensive method to deal with model uncertainties.
The paper tests stock return models and uses LSTM to predict stock returns.
problem Validating stock return models and predicting stock returns.
method Used Fama-French three-factor, four-factor, and five-factor models; also used LSTM model.
result Fama-French five-factor model shows better validity for stock returns.
Researchers review challenges in interpreting additive models, especially neural additive models.
problem Challenges in interpreting additive models, particularly neural additive models.
method Review of generalized additive models and discussion of nonidentifiability.
result Challenges in claiming interpretability or suitability for safety-critical applications of additive models.
Novel hybrid modeling combines ML and physics for real-time diagnosis.
problem Real-time diagnosis of complex systems.
method Combines machine learning and physics-based models to create reduced-order models.
result Generated models are two orders of magnitude simpler, improving efficiency.
CRS model improves ranking data modeling with theoretical guarantees.
problem Lack of rich, multimodal models for ranking data.
method Contextual Repeated Selection (CRS) model for multimodal ranking data.
result CRS model significantly outperforms existing methods in various ranking contexts.
Sigma models linked to Gross-Neveu models via quiver varieties.
problem Understanding the relationship between sigma models and Gross-Neveu models.
method Exploring the mathematical correspondence between sigma models and Gross-Neveu models, including their geometric and trigonometric/elliptic deformations.
result Sigma models are mathematically equivalent to Gross-Neveu models under certain conditions.
Interpretable machine learning has become a strong competitor for traditional black-box models. However, the possible loss of the predictive performance for gaining interpretability is often inevitable, putting practitioners in a dilemma of choosing between high accuracy (black-box models) and interpretability (interpr…
Simple models are preferred over complex models, but over-simplistic models could lead to erroneous interpretations. The classical approach is to start with a simple model, whose shortcomings are assessed in residual-based model diagnostics. Eventually, one increases the complexity of this initial overly simple model a…
Matryoshka hides secret models in a carrier model, achieving high capacity and robustness.
problem Stealing functionality of private ML data by hiding models in a carrier model.
method Parameter sharing approach exploiting the learning capacity of the carrier model.
result Hides a 26x larger secret model or 8 secret models in the carrier model.
Seq2Seq models speed up epidemic model predictions.
problem Complex epidemic models are computationally expensive.
method Used deep seq2seq models as surrogates for complex models.
result Surrogates predict scenarios up to several thousand times faster.
This work develops scalable model selection methods with fast update and selection.
problem Efficient model selection for large pools of candidate models.
method Isolated model embedding, which supports asymptotically fast update and selection.
result Standardized Embedder achieves competitive model selection performances.
Paper proposes BMPO to optimize policies using bidirectional models.
problem Model-based reinforcement learning's reliance on forward model accuracy.
method Develops BMPO using both forward and backward models for policy optimization.
result BMPO outperforms state-of-the-art methods in sample efficiency and asymptotic performance.
Copulas outperform marginal models in multivariate risk forecasting, reducing model risk by narrowing down the set of models.
problem Model risk in multivariate risk forecasting, especially during crises.
method Comprehensive empirical study comparing Copula-GARCH models with fixed marginals, copulas, or neither.
result Model risk is almost entirely due to copula choice, not marginal models.
BayesBlend blends multiple models' predictions for better insurance loss predictions.
problem Improving insurance loss predictions by combining multiple models.
method Pseudo-Bayesian model averaging, stacking, and hierarchical stacking.
result BayesBlend provides a user-friendly way to blend model predictions and estimate weights.
The paper identifies when larger models improve predictions and proposes a switcher model.
problem Understanding when larger models benefit from added complexity.
method Numerical studies on T5 architecture to analyze predictive uncertainty and model performance.
result Large models improve on examples where small models are uncertain, but not on certain examples.
Improved diffusion model generation speed with speculative sampling.
problem Generating samples from computationally expensive diffusion models.
method Extending speculative sampling to diffusion models, using fast draft models for candidate token generation.
result Significant speedup in generation, halving the number of function evaluations.
We propose a generalization of neural network sequence models. Instead of predicting one symbol at a time, our multi-scale model makes predictions over multiple, potentially overlapping multi-symbol tokens. A variation of the byte-pair encoding (BPE) compression algorithm is used to learn the dictionary of tokens that …
The paper extends statistical inference methods for black-box generative models.
problem Understanding and validating black-box generative models without access to their internal data.
method Develops model-level statistical inference tasks using generative model representations.
result Model-level representations are effective for multiple inference tasks.
PMM uses Bayesian inference to generate data from noisy approximations.
problem Creating flexible generative models for various data types.
method Bayesian inference and conjugate pairs of distributions.
result PMM achieves performance competitive with existing generative models.
Unified model improves sampling speed and quality.
problem Difficult to balance sampling speed and quality.
method Multistep Consistency Models combining consistency and diffusion models.
result Improved sampling quality with reduced steps.
In science and especially in economics, agent-based modeling has become a widely used modeling approach. These models are often formulated as a large system of difference equations. In this study, we discuss two aspects, numerical modeling and the probabilistic description for two agent-based computational economic mar…
Driven by an increasing need for model interpretability, interpretable models have become strong competitors for black-box models in many real applications. In this paper, we propose a novel type of model where interpretable models compete and collaborate with black-box models. We present the Model-Agnostic Linear Comp…
Combining models in appropriate ways to achieve high performance is commonly seen in machine learning fields today. Although a large amount of combinatorial models have been created, little attention is drawn to the commons in different models and their connections. A general modelling technique is thus worth studying …
MaxEnt Model Correction improves reinforcement learning model accuracy.
problem Improving reinforcement learning model accuracy and convergence.
method MaxEnt Model Correction (MoCo) procedure to correct model's next-state distributions.
result MoCoVI and MoCoDyna algorithms converge faster and effectively use approximate models.
Macroscopic price evolution models are commonly used for investment strategies. There are first promising achievements in defining microscopic agent based models for the same purpose. Microscopic models allow a deeper understanding of mechanisms in the market than the purely phenomenological macroscopic models, and thu…
Introduces model class selection to find sets of near-optimal models.
problem Finding sets of near-optimal models within multiple model collections.
method Generalizes model set selection framework to model class selection, using data splitting approaches.
result Shows that simpler, interpretable models can perform similarly to complex machine learning models.
Latent tree models are graphical models defined on trees, in which only a subset of variables is observed. They were first discussed by Judea Pearl as tree-decomposable distributions to generalise star-decomposable distributions such as the latent class model. Latent tree models, or their submodels, are widely used in:…
Gradient boosting enhances existing Mendelian models for genetic disease risk prediction.
problem Improving existing Mendelian models for genetic disease risk prediction.
method Combining gradient boosting with existing Mendelian models.
result Improved model outperforms both original and gradient boosting-only models.
Higher Gauge Flow Models integrate higher geometry and symmetries into Generative Flow Models.
problem Improving generative models' performance.
method Integrates L∞-algebra into Generative Flow Models, leveraging higher geometry and symmetries. result Substantial performance improvements on Gaussian Mixture Model datasets.
New data accumulation prevents model collapse in generative models.
problem Model collapse in generative models trained on their own outputs.
method Empirical study of language models, diffusion models, and variational autoencoders; analytically tractable framework for linear models.
result Accumulating synthetic data alongside real data avoids model collapse, preventing performance degradation.
BRAID fine-tunes diffusion models to optimize reward models in offline scenarios.
problem Combining generative modeling and model-based optimization in offline scenarios.
method Conservative fine-tuning of diffusion models using RL to optimize reward models.
result BRAID outperforms existing methods in offline data, avoiding invalid designs.