Deep neural networks are optimizable due to their multilayered structure.
problem Understanding why deep neural networks are easily optimizable despite their non-convex loss functions.
method Analysis of a spin glass model of deep neural networks using random matrix theory and algebraic geometry.
result The multilayered structure of deep neural networks leads to fewer stationary points, more clustered minima, and less severe tradeoffs between depth and width of minima.
The paper proves optimizability implies inequalities for sampling.
problem Optimizing functions via Gradient Flow and sampling from Gibbs measures.
method Gradient Flow and Lyapunov potentials to establish inequalities.
result Optimizability via Gradient Flow implies Poincaré and Log-Sobolev Inequalities.
New PAC-Bayesian bounds provide practical guarantees for neural networks.
problem Loose derandomization step in PAC-Bayesian bounds for deterministic models.
method Introduce disintegrated PAC-Bayesian bounds for deterministic models.
result Significant practical improvement over state-of-the-art bounds.
Noise Injection probes deep learning dynamics during training phases.
problem Understanding the learning mechanism of deep neural networks.
method Noise Injection Nodes (NINs) are used to perturb DNN architectures without altering the optimization algorithm.
result Distinct training phases are observed based on the scale of injected noise.
We propose a vector-valued regression problem whose solution is equivalent to the reproducing kernel Hilbert space (RKHS) embedding of the Bayesian posterior distribution. This equivalence provides a new understanding of kernel Bayesian inference. Moreover, the optimization problem induces a new regularization for the …
New method calibrates machine learning models with theoretical guarantees.
problem Lack of theoretical guarantees for recalibration in multiclass classification.
method PAC-Bayes analysis for generalization error in calibration.
result First optimizable upper bound for generalization error in calibration.
Agents acting in the natural world aim at selecting appropriate actions based on noisy and partial sensory observations. Many behaviors leading to decision mak- ing and action selection in a closed loop setting are naturally phrased within a control theoretic framework. Within the framework of optimal Control Theory, o…
A new scaling calculus helps design and initialize ReLU networks more effectively.
problem Optimizing the design and initialization of ReLU neural networks.
method Proposes a scaling constant for neural network layers and weights, relating it to optimizability.
result A network with a uniform scaling constant is easier to train, and the geometric mean of fan-in and fan-out is a better initialization variance.
Paper introduces a differentiable STFT for continuous window length optimization.
problem Optimizing window length in spectrograms for neural networks.
method Defines a differentiable short-time Fourier transform with continuous window length.
result Demonstrates improved performance in estimation and classification tasks.
This work optimizes alignment and uniformity of features on a hypersphere for better downstream performance.
problem Improving the performance of contrastive representation learning.
method Identifying and optimizing alignment and uniformity of features on a hypersphere.
result Directly optimizing alignment and uniformity leads to comparable or better performance than contrastive learning.
Paper proposes mnemonics framework for MCIL without forgetting.
problem Learning new concepts without forgetting previous ones in MCIL.
method Automatic framework using bilevel optimizations for parameterizing and optimizing exemplars.
result Mnemonics exemplars surpass state-of-the-art performance.
Revises Bayesian model averaging for foundation models.
problem Ensemble pre-trained and lightly-finetuned foundation models for improved classification performance.
method Introduces trainable linear classifiers and computationally cheaper model averaging scheme (OMA).
result Ensembled models can better predict on various datasets.
Many objects in the real world are difficult to describe by a single numerical vector of a fixed length, whereas describing them by a set of vectors is more natural. Therefore, Multiple instance learning (MIL) techniques have been constantly gaining on importance throughout last years. MIL formalism represents each obj…
SADCBO optimizes contextual variables by balancing relevance and cost.
problem Optimizing contextual variables with varying costs and unknown relevance.
method Adaptive selection of relevant contextual variables using sensitivity analysis and early stopping.
result Consistent improvement in optimization across various examples.
Paper proposes a data-free adversarial distillation method.
problem Tackles the challenge of model compression without real-world data.
method Introduces a novel adversarial distillation mechanism to craft a compact student model without any real-world data.
result Demonstrates comparable performance to data-driven methods and achieves state-of-the-art results in semantic segmentation.
This thesis tackles policy learning in large action spaces using on-policy and off-policy methods.
problem Challenges in learning with large action spaces, including inefficient exploration, sparse data, high-variance importance weights, and difficult optimization.
method Developed structured Bayesian methods for on-policy learning and structured direct methods for off-policy learning.
result Introduced meTS and dTS for on-policy learning, and sDM for off-policy learning, with regret guarantees and optimization strategies.
A new framework uses information theory to detect anomalies in images without labeled data.
problem Detect anomalies in images without labeled data.
method A direct objective function using information theory to maximize the distance between normal and anomalous data.
result The proposed framework significantly outperforms state-of-the-arts on multiple benchmark datasets.
Deep models store facts in geometric embeddings, not just associative memory.
problem Understanding how deep models store and utilize atomic facts.
method Identified geometric memory, contrasting with associative lookup.
result Geometric memory transforms hard reasoning into easy tasks.
New CNN approach detects cell nuclei with prior information.
problem Challenges in detecting cell nuclei due to image quality and morphology diversity.
method Develops SP-CNN with trainable shape prior layer to guide CNN learning.
result TSP-CNN outperforms state-of-the-art alternatives on challenging datasets.
A main challenge of data-driven sciences is how to make maximal use of the progressively expanding databases of experimental datasets in order to keep research cumulative. We introduce the idea of a modeling-based dataset retrieval engine designed for relating a researcher's experimental dataset to earlier work in the …
The paper introduces PD learning to improve deep learning theory.
problem Lack of theoretical understanding in deep learning model fitting and generalization.
method Proposes a PD learning framework to analyze optimization and generalization mechanisms of deep learning.
result Established theoretical guarantees on optimizability and derived generalization error bounds.
Optimized biopharmaceutical seed train design reduces variability and saves time.
problem Designing robust biotechnological processes with cell cultures.
method Coupling uncertainty-based upstream simulation and Bayes optimization using Gaussian processes.
result Optimized seed train design results in lower cell density variability and reduced process duration.
New bounds improve generalization in machine learning with high probability.
problem Erratic behavior of KL divergence limits practical applications.
method Replaced KL divergence with Wasserstein distance for better bounds.
result Proved high probability generalization bounds for i.i.d. and non-i.i.d. data.
New DG method minimizes barycentric alignment and reconstruction loss.
problem Improving domain generalization in machine learning.
method Introduces a new upper bound and WBAE algorithm.
result WBAE outperforms state-of-the-art DG algorithms.
This work connects neural network training to convex optimization via the NTK.
problem Understanding and optimizing neural network training via convex programs.
method Interpreting gated ReLU network as MKL, showing NTK equivalence, and improving weights.
result The NTK cannot perform better than the optimal MKL kernel on the training set.
Efficiently verifies neural networks by handling neuron splits, improving speed and accuracy.
problem Handling neuron split constraints in incomplete neural network verification.
method β-CROWN, which optimizes parameters β to encode neuron splits and uses them in bound propagation.
result β-CROWN significantly speeds up verification while maintaining high accuracy.
Develops Fourier analysis for SDR, simplifying dimension reduction problems.
problem Finding a reduced subspace for high-dimensional data.
method Formulates SDR as a minimization problem in the Fourier domain, introducing a penalty for support distortion.
result An algorithm for finding a reduced subspace using Fourier transforms and penalties.
Improves Bayesian inference for deep models to better approximate posterior distributions.
problem Bayesian deep learning struggles with intractable posterior distributions, leading to overconfident predictions.
method Uses variational inference to approximate posterior distributions, proposing a unified view and improving inference for deep Gaussian processes.
result Variational inference can provide a lower bound for marginal likelihood, facilitating model selection and optimization.