Sharp bounds on neural network approximation rates and widths.
problem Estimating approximation rates, metric entropy, and n-widths of shallow neural networks.
method Introducing smoothly parameterized dictionaries and providing upper and lower bounds.
result Sharp bounds on approximation rates, metric entropy, and n-widths for neural networks with various activation functions.
Paper proposes a new uncertainty measure for active learning in neural networks.
problem Efficiently selecting informative data points in limited labeled data scenarios.
method BalEntAcq, a new uncertainty measure based on balanced entropy, approximated by Beta distributions.
result BalEntAcq outperforms existing uncertainty measures in active learning.
Proposes an active RBI framework using Rényi information measures for more informed decision-making.
problem Optimal latent variable estimates in real-time settings with streaming noisy observations.
method Unified inference and query selection steps through Rényi entropy and α-divergence; new objective called Momentum for exploration.
result Analytically demonstrates superior performance compared to conventional methods like mutual information.
ESPRESSO segments time-series data for better human activity recognition.
problem Segmenting high-dimensional time-series data for applications like HAR.
method ESPRESSO combines entropy and shape analysis for multi-dimensional time-series segmentation.
result ESPRESSO outperforms four state-of-the-art methods across seven datasets.
New method combines active and semi-supervised deep learning for high accuracy with minimal labeled data.
problem Expensive label generation in machine learning.
method Combines active and semi-supervised deep learning, using expectation maximization and active learning.
result Achieves high accuracy (2.06% for 300 labels, 1.06% for 1000 labels) with minimal labeled data.
Bayesian active learning method improved for censored regression data.
problem Challenges in estimating BALD for censored regression data.
method Derived entropy and mutual information for censored distributions, developed C-BALD objective, proposed novel modelling approach. result Demonstrated C-BALD outperforms other methods in censored regression. Optimal policy reduces entropy in preference learning.
problem Learning user preferences via choice-based queries.
method Parametric model with Bayesian prior, greedy policy for entropy reduction.
result Greedy policy achieves linear lower bound on entropy reduction.
This research uses activation entropy to improve speech recognition under unseen data conditions.
problem Performance degradation in speech recognition systems under unseen data conditions.
method Estimates network activation entropy to measure data variability and select data for unsupervised model adaptation.
result Improves speech recognition performance under unseen data conditions.
Adversarial Sampling improves active learning by generating high entropy synthetic samples.
problem Improving active learning performance through better sample selection.
method Generative Adversarial Network (GAN) based approach to generate high entropy synthetic samples, then select similar real samples for training.
result ASAL outperforms random sample selection and other active learning methods across multiple datasets.
New insights into neural network training efficiency.
problem Understanding the optimal initialization for deep neural networks.
method Exploring the edge of chaos and saturation of tanh activation function.
result The line of uniformity in phase space intersects the edge of chaos, indicating saturation begins to hinder training efficiency.
FedMAX tackles activation divergence in FL, improving accuracy and efficiency.
problem Activation divergence in Federated Learning due to non-IID data.
method Introduced a prior based on maximum entropy to minimize information about per-device activation vectors and make similar classes more alike.
result Significantly more similar activation vectors across multiple devices, leading to better accuracy and efficiency.
Sharp lower bounds on shallow neural networks' approximation rates are derived.
problem The efficiency of shallow neural networks in approximating functions.
method Lower bounding the L2-metric entropy and Kolmogorov n-widths of the convex hull of neural network basis functions. result Sharp lower bounds on the approximation rates for shallow neural networks are provided.
Study shows MSE with sigmoid can match SCE in classification tasks, especially with noisy data.
problem Inconsistent errors in neural network classification tasks.
method Introduced Output Reset algorithm to use MSE with sigmoid activation.
result MSE with sigmoid activation achieves comparable accuracy and convergence rates to Softmax Cross-Entropy, especially in noisy data scenarios.
New methods improve Bayesian inference and decision-making in online learning.
problem Current Bayesian deep learning does not fully utilize joint predictives for sequential decision-making.
method Proposes new evaluation settings for active learning and active sampling, focusing on marginal and joint cross-entropies.
result Initial experiments suggest challenges in applying current BDL inference techniques in high-dimensional spaces.
Improves GAN training by guiding the discriminator to have more diverse binary activation patterns.
problem Stability and convergence issues in GAN training.
method Binarized Representation Entropy (BRE) regularization to guide the discriminator's model capacity allocation.
result Improves GAN training stability and convergence speed, higher sample quality, and higher classification accuracy.
Softmax cross-entropy optimizes mutual information in neural networks.
problem Understanding the relationship between mutual information and classification neural networks.
method Demonstrated that optimizing softmax cross-entropy maximizes mutual information between inputs and labels.
result Softmax cross-entropy can approximate mutual information and highlight relevant image regions.
This paper addresses the problem of active learning of a multi-output Gaussian process (MOGP) model representing multiple types of coexisting correlated environmental phenomena. In contrast to existing works, our active learning problem involves selecting not just the most informative sampling locations to be observed …
Optimal observation selection for complex hidden hypotheses.
problem Diagnosis of complex hidden hypotheses using limited observations.
method Active diagnosis through selection of most informative observations based on past results.
result An implication model that predicts future outcomes based on past observations, selecting the most informative next observation.
The article explores how organisms and machines learn and recognize the world using Bayesian inference and thermodynamics.
problem Understanding how organisms and machines learn and recognize the world.
method Introducing a thermodynamic view of the Bayesian brain hypothesis, using a simple generative model of spiking neural populations.
result The process of Bayesian inference can be quantified using entropy, revealing the perceptual capacity of neural activity.
Wavelet analysis reveals financialization effects on oil-food price correlation.
problem Investigating the correlation between oil and food prices and their determinants.
method Wavelet analysis and energy-based measures to differentiate high and low frequency movements.
result Significant local correlation between food and oil is due to financialization and emerging economies' demand.
This paper proposes an improved active learning method using classification trees.
problem Reducing the size of training sets while maintaining high accuracy in supervised learning.
method A wrapper active learning method using a classification tree to sub-sample from low-entropy regions.
result The proposed method constructs accurate classification models even with severely restricted labeled data.
A new active learning method for Gaussian process models.
problem Efficiently exploring unbounded state spaces for accurate models.
method Maximizes mutual information with respect to a bounded region using model predictive control.
result Our method yields a better model within the region of interest than entropy-based methods.
IWeS selects examples by entropy-based importance sampling for subset selection.
problem Efficiently selecting examples for model training in batch settings.
method IWeS uses importance sampling based on model entropy to select examples.
result IWeS outperforms other subset selection algorithms on seven datasets.
Crowdsourcing and active learning reduce manual annotation in social media event classification.
problem Manual annotation is time-consuming and resource-intensive in social media event classification.
method Crowdsourcing pipeline combined with active learning strategies.
result Active learning strategies help reduce the number of tweets needed for classification.
We study the activity, i.e., the number of transactions per unit time, of financial markets. Using the diffusion entropy technique we show that the autocorrelation of the activity is caused by the presence of peaks whose time distances are distributed following an asymptotic power law which ultimately recovers the Pois…
Deep networks avoid bad local minima with no bad valleys.
problem Finding sub-optimal local minima in deep neural networks.
method Analyzing over-parameterized networks with specific activation and loss functions.
result No bad local valleys, implying no sub-optimal strict local minima.
Bal-PM reduces preference labeling costs for LLMs.
problem Efficiently acquiring human feedback for preference modeling in large language models.
method Bayesian Active Learning with entropy maximization in feature space.
result Bal-PM reduces the number of required preference labels by 33% to 68%.
Contextual policy search allows adapting robotic movement primitives to different situations. For instance, a locomotion primitive might be adapted to different terrain inclinations or desired walking speeds. Such an adaptation is often achievable by modifying a small number of hyperparameters. However, learning, when …
The ultimate goal of optimization is to find the minimizer of a target function.However, typical criteria for active optimization often ignore the uncertainty about the minimizer. We propose a novel criterion for global optimization and an associated sequential active learning strategy using Gaussian processes.Our crit…
Cognitive abstraction metrics predict deep neural networks' generalization ability.
problem Understanding deep neural networks' generalization ability.
method Defined Cognitive Neural Activation (CNA) metric based on information complexity and activation patterns.
result CNA is highly predictive of generalization ability, outperforming other metrics.
A new mutual information lower bound for multimodal regression active learning.
problem Lack of effective acquisition functions for multimodal regression active learning.
method Introduces a Two-Index framework for separating epistemic and aleatoric sources of uncertainty, deriving MI-LB as a closed-form approximation.
result MI-LB consistently outperforms baselines on multimodal regression tasks.
The article presents a new entropy model for assessing stock market interest.
problem Assessing investor interest and market sentiment in exchange-traded securities.
method Intrinsic entropy model using actual trading data, without exogenous factors.
result Empirical evidence supports the model's ability to predict trading activity.
Enhances activity recognition in wearable computing with context awareness and uncertainty quantification.
problem Context-dependent activity recognition and unknown contexts in wearable computing.
method Developed the α-{eta} network coupled with uncertainty quantification (UQ) based on maximum entropy.
result Improved accuracy and F-score by 10% through high-level context identification.
Personalized activity recognition improves performance for diverse users.
problem Poor performance of impersonal algorithms for individual users.
method Personalized activity recognition using deep embeddings from a fully convolutional neural network with triplet loss.
result Novel subject triplet loss provides the best performance overall.
A method for learning from unlabeled time-series data using temporal smoothing and entropy maximization.
problem Learning from unlabeled time-series data efficiently and accurately.
method Training a feedforward neural network with two objectives: temporal smoothing and entropy maximization.
result The method extracts slowly evolving information from time-series data, filtering out noise.
Improved CNN for HCCR with new loss function and ranking method.
problem Loss of inter-class information in traditional CNN models for HCCR.
method Combining cross entropy with a new similarity ranking function (Average variance similarity) as loss function.
result New loss function (SoftMax cross entropy with Average variance similarity) achieves highest accuracy in HCCR.
GOIMDA selects inputs to maximize expected influence on a goal functional, reducing data acquisition needs.
problem Challenges in active data acquisition for learning and optimization tasks in deep neural networks.
method GOIMDA uses inverse curvature and goal gradient to select inputs maximizing expected influence on a specified goal functional.
result GOIMDA achieves target performance with fewer labeled samples or function evaluations compared to baselines.
The paper analyzes how well classes are separated in neural network feature space.
problem Understanding class separability in neural network feature space.
method Theoretical analysis of intra-class and inter-class distances in feature space.
result A lower bound for the probability of inter-class distance being greater than intra-class distance as a function of loss value.
Bayesian active learning improves holistic educational assessments.
problem Gap between holistic CJ and criterion-based rubrics in education.
method Extends Bayesian CJ to handle multiple LO components, using entropy-based active learning.
result Enhanced predictive rankings with uncertainty estimates and quantified assessor agreement.
We adapt Shapley values to explain model uncertainty, connecting it to information theory.
problem Explaining uncertainty in model predictions.
method Adapted Shapley value framework to quantify feature contributions to predictive uncertainty.
result Deep connections between Shapley values and information theory quantities.
NM-PPG optimizes adaptive feature acquisition in POMDPs for better predictions.
problem Optimizing adaptive feature acquisition in prediction problems with costly features.
method Non-myopic pathwise policy gradients (NM-PPG) with continuous relaxation and straight-through rollout.
result NM-PPG outperforms state-of-the-art AFA methods on synthetic and real-world datasets.
Gradient flow in softmax models tends to produce low-entropy outputs.
problem Understanding the training dynamics of softmax-based models.
method Analysis of gradient flow dynamics in the value-softmax model.
result Gradient flow drives optimization towards low-entropy solutions.
Gaussian Mixture Models (GMM) have found many applications in density estimation and data clustering. However, the model does not adapt well to curved and strongly nonlinear data. Recently there appeared an improvement called AcaGMM (Active curve axis Gaussian Mixture Model), which fits Gaussians along curves using an …
Study predicts price predictability in ultra-high frequency financial data using entropy tests.
problem Tackles predictability of ultra-high frequency financial data.
method Develops statistical tests based on Shannon entropy and Kullback-Leibler divergence to analyze predictability.
result Degree of randomness increases with aggregation level in transaction time.
A new active learning method considers both uncertainty and diversity to minimize labeling and decision costs.
problem Classical AL approaches fail to capture data distribution in unlabeled data, leading to mislabeling of outliers.
method CBAL considers classification uncertainty and instance diversity, using a min-max approach to minimize labeling and decision costs.
result Extensive experiments show CBAL outperforms state-of-the-art AL approaches.
Proves eventological H-theorem complementing Boltzmann's.
problem Mathematical justification for the second law of eventology.
method Proves eventological H-theorem using relative entropy.
result Mathematical support for Gibbs and anti-Gibbs distributions.
We consider the problem of learning convex aggregation of models, that is as good as the best convex aggregation, for the binary classification problem. Working in the stream based active learning setting, where the active learner has to make a decision on-the-fly, if it wants to query for the label of the point curren…
Bayesian search optimizes exploration of feasible solutions under expensive constraints.
problem Identifying feasible solutions in computationally expensive constraint spaces.
method Bayesian models with an acquisition function for efficient exploration and exploitation.
result The proposed acquisition function improves the prediction of feasibility.