Active learning method balances bias and variance under class imbalance.
problem Active learning under label shift when class proportions differ.
method Mediated Active Learning under Label Shift (MALLS) using a 'medial distribution'.
result MALLS reduces asymptotic sample complexity under arbitrary label shift.
Study long-only minimum variance portfolio in one-factor market with arbitrary sign betas.
problem Characterize the long-only minimum variance portfolio in a one-factor market with mixed-sign betas.
method Explicit solution for long-only minimum variance portfolio, explicit characterization of active set, asymptotic analysis in high-dimensional regime.
result Proportion of active assets in LOMV portfolio converges to F(β∗) in high-dimensional regime, with rate O(F(0)1/3) when F(0)>0. Normalization techniques play an important role in supporting efficient and often more effective training of deep neural networks. While conventional methods explicitly normalize the activations, we suggest to add a loss term instead. This new loss term encourages the variance of the activations to be stable and not va…
EVA adapts LoRA for faster, more efficient fine-tuning.
problem Fast and efficient fine-tuning of large models for specific tasks.
method EVA uses directions capturing most activation variance for initialization, maximizing gradient signal and reducing parameters.
result EVA achieves faster convergence and higher average scores across tasks, reducing parameters.
Active learning aims to train a classifier as fast as possible with as few labels as possible. The core element in virtually any active learning strategy is the criterion that measures the usefulness of the unlabeled data based on which new points to be labeled are picked. We propose a novel approach which we refer to …
REPAIR mitigates variance collapse to enable linear interpolation between SGD solutions.
problem Linear interpolation between SGD solutions is difficult due to variance collapse in permuted activations.
method REPAIR rescales preactivations of interpolated networks to mitigate variance collapse.
result 60%-100% relative barrier reduction across various architectures and tasks.
The paper tackles mean-variance analysis in Bayesian optimization under uncertainty.
problem Optimizing decisions in uncertain environments considering trade-offs between average and variance of risk.
method Developed bounds for mean and variance risk measures in Gaussian Process models and proposed AL algorithms for multi-task, multi-objective, and constrained optimization scenarios.
result Proposed AL algorithms effectively address the mean-variance trade-off in uncertain optimization scenarios.
Depth uncertainty networks don't improve with bias correction, contrary to expectations.
problem Improving performance in active learning with overparameterised models like NNs.
method Depth uncertainty networks, compared to underparameterised models, show no improvement in performance with bias correction.
result Depth uncertainty networks do not improve with bias correction, unlike underparameterised models.
Deep Learning has revolutionized vision via convolutional neural networks (CNNs) and natural language processing via recurrent neural networks (RNNs). However, success stories of Deep Learning with standard feed-forward neural networks (FNNs) are rare. FNNs that perform well are typically shallow and, therefore cannot …
Improves active learning efficiency by warping input space based on observed outputs.
problem Insensitivity of Gaussian process uncertainty to actual observations.
method Input warping with learned monotone reparameterization to adjust acquisition function behavior.
result Significantly improved sample efficiency across various benchmarks, especially in non-stationary conditions.
Improved EXACT strategy reduces GNN memory consumption and runtime.
problem Efficiently training large-scale GNNs with reduced memory usage.
method Block-wise quantization of intermediate activation maps with improved variance minimization.
result Further reduction in memory consumption (>15%) and runtime speedup (5%) with similar performance trade-offs.
New method avoids failures in physics-constrained systems using active learning.
problem Handling fatal failures in systems governed by physics constraints.
method Develops a novel active learning method that considers implicit physics constraints.
result Achieves zero-failure in composite fuselage assembly process without explicit failure regions.
Neural SDEs reduce variance in stochastic simulations.
problem Efficiency of Monte Carlo simulations in finance.
method Use neural SDEs with control variates parameterized by neural networks.
result Prove optimality conditions for variance reduction in SDEs with infinite activity.
PPAT uses predictions to improve risk estimation in active testing.
problem Exploiting informative predictions from black-box models for efficient risk estimation.
method Combines LURE estimator with prediction-powered control variate.
result PPAT outperforms existing methods in risk estimation and uncertainty quantification.
New theory explains how overparametrized neural networks generalize well without bias-variance trade-off.
problem Overparametrized neural networks generalize well despite classical bias-variance trade-off.
method Nonasymptotic generalization theory for two-layer neural networks with ReLU activation, incorporating scaled variation regularization.
result Prediction bounds for all network widths reproduce the double descent phenomenon, and overparametrized models are nearly minimax optimal.
Study adapts AI research methods to analyze image augmentation impacts on neural network operations.
problem Understanding how image augmentation affects neural network performance and sensitivity.
method Adapted treatment-control paradigm, uses variance decomposition, Sobol indices, and Shapley values for sensitivity analysis.
result Visualizes and quantifies sensitivity to different image augmentation parameters.
LSQ+ improves quantization of neural nets with Swish activations, achieving state-of-the-art results.
problem Quantization of neural nets with Swish activations, especially negative activations, leads to significant performance loss.
method Introduces learnable scale and offset parameters for asymmetric quantization, and uses MSE-based initialization for quantization parameters.
result Significantly outperforms LSQ for low-bit quantization of neural nets with Swish activations, achieving up to 5.6% gain with W2A2 quantization of EfficientNet-B0.
Improved LLM pre-training performance through better weight and variance control.
problem Improper weight and variance control in LLM pre-training affects downstream task performance.
method Introduced Layer Index Rescaling (LIR) and Target Variance Rescaling (TVR) techniques.
result Substantial improvements in downstream task performance (up to 4.6%) and reduced extreme activation values.
Bayesian adaptive designs can be biased by active learning, especially with misspecified models.
problem Active learning bias in Bayesian adaptive experimental designs.
method Analysis of linear and preference learning models, empirical testing.
result Model misspecification and noise influence active learning bias in Bayesian designs.
Before training a neural net, a classic rule of thumb is to randomly initialize the weights so the variance of activations is preserved across layers. This is traditionally interpreted using the total variance due to randomness in both weights \emph{and} samples. Alternatively, one can interpret the rule of thumb as pr…
ActiveCQ improves causal quantity estimation with active learning and Gaussian Processes.
problem Estimating causal quantities requires large datasets, which are costly.
method Unified framework using Gaussian Processes and conditional mean embeddings for distribution estimation. Derived principled acquisition strategies based on information gain and total variance reduction.
result Framework significantly outperforms baselines in sample efficiency across various causal quantities.
In training speech recognition systems, labeling audio clips can be expensive, and not all data is equally valuable. Active learning aims to label only the most informative samples to reduce cost. For speech recognition, confidence scores and other likelihood-based active learning methods have been shown to be effectiv…
New method reduces PDE surrogate model training costs by selectively acquiring time steps.
problem High computational cost of generating training data for PDE surrogate models.
method STAP (Selective Time-Step Acquisition for PDEs) framework that acquires only important time steps.
result Demonstrated effectiveness on several benchmark PDEs, reducing training costs.
Paper improves confidence intervals and variance estimation for deep learning models.
problem Improving confidence intervals and variance estimation in deep learning models.
method Residual-based framework for conditional variance estimation; robust bootstrap procedure for confidence intervals.
result First non-asymptotic bounds for variance estimation using ReLU networks.
Bayesian optimization improves policy search in reinforcement learning.
problem Finding optimal policies with high variance estimates from random samples.
method Develops an algorithm combining Bayesian optimization and policy gradients.
result Improves sample complexity and reduces variance in empirical evaluations.
Active testing reduces label costs for efficient model evaluation.
problem Real-world applications require expensive test labels, disconnecting from existing model evaluation methods.
method Derives acquisition strategies to select test points efficiently, addressing label bias and variance.
result Active testing improves model evaluation efficiency without sacrificing accuracy.
GAPA method provides efficient uncertainty quantification for pretrained networks.
problem Reliable uncertainty estimates for pretrained models are challenging.
method Post-hoc Gaussian Process Activations (GAPA) method that shifts Bayesian modeling from weights to activations.
result GAPA method provides efficient uncertainty quantification without altering the backbone's predictions.
New initialization schemes preserve fractional moments of weights in deep networks, improving training and test performance.
problem Heavy-tailed distribution of stochastic gradients in DNNs during training.
method Developed initialization schemes that preserve any given fractional moment of order s < 2 over layers for various activations.
result The network output admits a heavy-tailed distribution with finite moments, improving training and test performance.
We propose and investigate new complementary methodologies for estimating predictive variance networks in regression neural networks. We derive a locally aware mini-batching scheme that result in sparse robust gradients, and show how to make unbiased weight updates to a variance network. Further, we formulate a heurist…
The study analyzes ETFs' portfolio optimization and tail-risk management.
problem Analyzing the performance of actively managed ETFs in managing risk and diversification.
method Daily Bloomberg data for 30 funds, evaluating various strategies under long-only and long-short constraints.
result Tangency-type portfolios generally outperform buy-and-hold benchmarks, while minimum-variance and CVaR-minimizing portfolios sacrifice upside for downside control.
Active inference framework improves U-statistic estimation efficiency.
problem Costly acquisition of labels for U-statistics. method Active inference framework with optimal sampling rule.
result Substantial gains in estimation efficiency over baseline methods.
LCW reduces activation shift in neural networks, improving training efficiency and generalization.
problem Activation shift in neural networks leading to non-zero mean preactivation values.
method Linearly constrained weights (LCW) to reduce activation shift in fully connected and convolutional layers.
result LCW resolves the vanishing gradient problem and improves generalization of neural networks.
Obtaining labels can be costly and time-consuming. Active learning allows a learning algorithm to intelligently query samples to be labeled for efficient learning. Fisher information ratio (FIR) has been used as an objective for selecting queries in active learning. However, little is known about the theory behind the …
DiAL uses Bayesian Dirichlet random fields for active learning with sparse labels.
problem Active learning with limited labeled data.
method Bayesian Dirichlet random field for feature-conditional class probabilities, calibrating with graph Laplacian.
result Competitive performance in low-label rate graph learning tasks.
New active learning methods for Gaussian process improve predictive modeling of composite fuselage.
problem Improving predictive modeling of composite fuselage with limited training samples and uncertainties.
method Proposed two new active learning algorithms for Gaussian process considering uncertainties.
result The proposed approach realizes better prediction performance for automatic shape control of composite fuselage.
Develops active learning for Jump Gaussian Process models.
problem Optimizing experimental designs and steering data acquisition in complex systems.
method Active learning of piecewise Jump Gaussian Process (Jump GP) models, accounting for model bias.
result Demonstrates the importance of accounting for model bias in Jump GP models.
We analyze the joint probability distribution on the lengths of the vectors of hidden variables in different layers of a fully connected deep network, when the weights and biases are chosen randomly according to Gaussian distributions, and the input is in {−1,1}N. We show that, if the activation function φ sat…
Improves Active Learning fairness and comparability across domains.
problem Inconclusive Active Learning research due to domain-specific results.
method Develops a fair comparison framework and oracle algorithm.
result Empirical results rank 6 algorithms across 3 domains.
AM-PPI uses multiple predictors to reduce label cost in healthcare AI.
problem Reduces label cost in post-deployment monitoring of healthcare AI.
method Combines model predictions with a small labeled sample, routing each instance to a cost-appropriate subset of predictors.
result Produces narrower confidence intervals than single-predictor methods.
The convergence speed of stochastic gradient descent (SGD) can be improved by actively selecting mini-batches. We explore sampling schemes where similar data points are less likely to be selected in the same mini-batch. In particular, we prove that such repulsive sampling schemes lowers the variance of the gradient est…
Paper proposes a new activation function to reduce overfitting and large weight update issues.
problem Overfitting and large weight update problems in neural networks.
method Introduces a new activation function called Thresholded Exponential Rectified Linear Units (TERELU).
result TERELU shows better performance in reducing overfitting and large weight update issues compared to other activation functions.
PALS extends PAL for optimizing stochastic simulators efficiently.
problem Optimizing stochastic simulators with high output variance and expensive evaluations.
method Bayesian optimization with probabilistic models, extending PAL for stochastic settings.
result PALS outperforms other methods in optimizing stochastic simulators.
Ensembles improve classifier performance by reducing bias, not variance.
problem Improving classifier performance through ensemble methods.
method Extended bias-variance decomposition for classification tasks, introducing dual reparameterization.
result Ensembling reduces bias in classifiers, contrary to the traditional view.
Proposes a stratified sampling method for high-dimensional models using neural active manifolds.
problem Uncertainty propagation in computationally expensive models with many inputs.
method Neural active manifolds for nonlinear dimensionality reduction, followed by stratification in the reduced space.
result Effective variance reduction in high-dimensional models using stratified sampling.
We identify and study two common failure modes for early training in deep ReLU nets. For each we give a rigorous proof of when it occurs and how to avoid it, for fully connected and residual architectures. The first failure mode, exploding/vanishing mean activation length, can be avoided by initializing weights from a …
We consider in this paper the problem of optimal experiment design where a decision maker can choose which points to sample to obtain an estimate β^ of the hidden parameter β⋆ of an underlying linear model. The key challenge of this work lies in the heteroscedasticity assumption that we make, meaning that…
Observing prices of European put and call options, we calibrate exponential Lévy models nonparametrically. We discuss the efficient implementation of the spectral estimation procedures for Lévy models of finite jump activity as well as for self-decomposable Lévy models. Based on finite sample variances, confidence inte…
Introduces a new Lévy process for modeling illiquid markets.
problem Modeling dynamic of assets in illiquid markets.
method Introduces Variance Gamma++ process, a new Lévy process, and provides efficient path simulation algorithms.
result Efficient pricing formula and parameter estimation for European options.