A method to reduce memory usage in deep learning models by adding inducing weights.
problem Memory inefficiency in Bayesian neural networks and deep ensembles.
method Augmenting the weight matrix with inducing weights and using Matheron's conditional Gaussian sampling rule.
result Reduces parameter size to 24.3% of a single neural network while maintaining competitive performance.
Weight Decay induces low-rank weight matrices in neural networks, improving generalization.
problem Improving generalization in neural networks.
method Training ReLU NN with Weight Decay and Stochastic Gradient Descent.
result The weight matrix of a trained NN is approximately rank-two.
Global inducing points improve Bayesian neural network performance.
problem Improving Bayesian neural network performance.
method Adapting correlated approximate posterior to all layers in a Bayesian neural network and deep Gaussian processes using learned global inducing points.
result State-of-the-art performance on CIFAR-10 (86.7%) without data augmentation or tempering.
Any Sasakian structure can be closely mimicked by embeddings into weighted spheres.
problem Approximating Sasakian structures on closed manifolds.
method Using CR embeddings into weighted Sasakian spheres and strengthening previous approximation results.
result Sasakian structures can be approximated in the Cq-norm by embeddings into weighted Sasakian spheres. Gradient descent with random weights in linear regression analyzed for various noise types.
problem Analyzing the impact of random noise on gradient descent in linear regression.
method Gradient descent with randomly weighted data points, various weighting distributions, geometric moment contraction.
result Characterization of implicit regularization and non-asymptotic convergence bounds.
Replacing MSE with f-divergence in diffusion models improves robustness under data contamination.
problem Improving robustness of diffusion models under data contamination.
method Replacing MSE with f-divergence in diffusion models.
result Empirical improvement in performance under data contamination.
In recent years, deep neural networks (DNNs) have been applied to various machine leaning tasks, including image recognition, speech recognition, and machine translation. However, large DNN models are needed to achieve state-of-the-art performance, exceeding the capabilities of edge devices. Model reduction is thus nee…
We investigate the relation between weighted quasi-metric Spaces and Finsler Spaces. We show that the induced metric of a Randers space with reversible geodesics is a weighted quasi-metric space.
Deep Bayesian neural nets can use simpler weight approximations without sacrificing performance.
problem The need for complex weight posterior approximations in deep Bayesian neural networks.
method Theoretical and empirical analysis of mean-field variational inference in deep networks.
result Mean-field variational weight posteriors in deep networks can induce similar function-space distributions as complex approximations in shallower networks.
Enhances trading signals using image analysis and weighted moving averages.
problem Improving price trend trading strategies in financial markets.
method Image-induced importance weights applied to weighted moving averages of trading signals.
result Significant enhancement of price trend trading signals with improved portfolio selection.
We extend the notion of intersection graphs for knots in the theory of finite type invariants to string links. We use our definition to develop weight systems for string links via the adjacency matrix of the intersection graph, and show that these weight systems are related to the weight systems induced by the Conway a…
Characterizes inductive bias in multi-channel linear CNNs with bounded weight norm.
problem Understanding the inductive bias in multi-channel linear convolutional networks.
method Function space characterization and empirical testing of gradient descent.
result The inductive bias depends on the number of output channels for multi-channel inputs but not for single-channel inputs.
Learning representation on graph plays a crucial role in numerous tasks of pattern recognition. Different from grid-shaped images/videos, on which local convolution kernels can be lattices, however, graphs are fully coordinate-free on vertices and edges. In this work, we propose a Gaussian-induced convolution (GIC) fra…
Corrects distribution shift in target shift scenarios using importance weighting.
problem Analyzes importance weighting for correcting distribution shift under target shift.
method Analyzed importance-weighted kernel ridge regression under target shift.
result Shows that importance weighting corrects the train-test mismatch without altering input-space complexity.
Optimal Euclidean structure minimizes energy in weighted toroidal graphs.
problem Finding the optimal Euclidean structure for weighted toroidal graphs.
method Minimizing Dirichlet energy over all possible Euclidean structures and realizations within a fixed homotopy class.
result The optimal Euclidean structure induces a weighted Delaunay decomposition.
This work aims at solving the problems with intractable sparsity-inducing norms that are often encountered in various machine learning tasks, such as multi-task learning, subspace clustering, feature selection, robust principal component analysis, and so on. Specifically, an Iteratively Re-Weighted method (IRW) with so…
HALO learns to prune neural networks by adaptively shrinking weights.
problem Sparsity and model size in deep neural networks.
method Bayesian hierarchical models and trainable parameters for adaptive sparsification.
result HALO learns to create highly sparse networks with significant performance gains.
In our previous paper, we discussed the hyperbolization of the configuration space of n(> 4) marked points with weights in the projective line up to projective transformations. A variation of the weights induces a deformation. It was shown that this correspondence of the set of the weights to the Teichmüller space when…
SURF steers scalarization weights to uniformly traverse the Pareto front.
problem Non-uniform coverage of the Pareto front when using scalarization weights.
method Geometric analysis and CDF mapping to select weights for uniform coverage.
result SURF converges to uniform Pareto front coverage under provable conditions.
Study on feature learning dynamics in infinite-depth neural networks, focusing on ResNets.
problem Understanding how features evolve during training in deep neural networks, especially in the large-depth limit.
method Conditional Gaussian representations and SDE system with decoupled backward weights.
result Depth-induced suppression of forward-backward coupling in infinite-depth networks, leading to a decoupled forward-backward SDE system.
Let M be a compact connected oriented n−1 dimensional manifold without boundary. In this work, shape space is the orbifold of unparametrized immersions from M to Rn. The results of \cite{Michor118}, where mean curvature weighted metrics were studied, suggest incorporating Gauß curvature weights in the …
Regularization can induce grokking in neural networks, improving generalization.
problem Delayed generalization following overfitting in neural networks.
method Demonstrates that gradient descent with small regularization of model properties induces grokking.
result Regularization can induce grokking, extending previous work on weight decay.
We investigate deep Bayesian neural networks with Gaussian weight priors and a class of ReLU-like nonlinearities. Bayesian neural networks with Gaussian priors are well known to induce an L2, "weight decay", regularization. Our results characterize a more intricate regularization effect at the level of the unit activat…
In the present paper, we study deformations of polar weighted homogeneous polynomials which are also polar weighted homogeneous polynomials. We describe a round handle decomposition of the Milnor fibration of a deformation of a polar weighted homogeneous polynomial concretely and give the number of round handles by the…
Importance weighted variational inference (Burda et al., 2015) uses multiple i.i.d. samples to have a tighter variational lower bound. We believe a joint proposal has the potential of reducing the number of redundant samples, and introduce a hierarchical structure to induce correlation. The hope is that the proposals w…
Cautious Weight Decay modifies weight decay for better optimization.
problem Improving optimization in deep learning models.
method Applies weight decay selectively based on parameter sign alignment.
result Consistently improves model performance across various tasks and scales.
B-cos transforms improve neural network interpretability by aligning weights.
problem Improving interpretability of deep neural networks.
method Replacing linear transforms with B-cos transforms that promote weight-input alignment during training.
result B-cos transforms lead to highly interpretable and task-relevant linear summaries of neural network computations.
Paper investigates rigidity phenomena for weighted Ricci curvature bounds with Laplacian comparison theorem.
problem Investigating rigidity phenomena for weighted Ricci curvature bounds.
method Derived comparison geometric estimates and generalized for non-symmetric Laplacian.
result Obtained rigidity results for Laplacian comparison theorem, diameter comparisons, and volume comparisons.
Algorithmic approaches endow deep learning systems with implicit bias that helps them generalize even in over-parametrized settings. In this paper, we focus on understanding such a bias induced in learning through dropout, a popular technique to avoid overfitting in deep learning. For single hidden-layer linear neural …
The paper analyzes prediction error in nonstationary settings using weighted risk minimization.
problem Prediction under distribution drift and nonstationary conditions.
method General decomposition of excess risk into learning and drift terms, proving oracle inequalities under mixing conditions.
result Oracle inequalities for the learning error, providing bounds that hold uniformly over arbitrary weight classes.
SGD and weight decay encourage neural networks to learn low-rank weight matrices.
problem The bias of SGD towards low-rank weight matrices in neural networks.
method The study investigates the effect of SGD and weight decay on the rank of weight matrices in neural networks, both theoretically and empirically.
result Training with SGD and weight decay induces a bias towards rank minimization in weight matrices, which becomes more pronounced with smaller batch sizes and stronger weight decay.
Prior-weighted logistic regression has become a standard tool for calibration in speaker recognition. Logistic regression is the optimization of the expected value of the logarithmic scoring rule. We generalize this via a parametric family of proper scoring rules. Our theoretical analysis shows how different members of…
Bayesian framework improves minority class performance in class-imbalanced data.
problem Class imbalance in predictive toxicology models.
method Weighted likelihood approach modifying likelihood function weights inversely proportional to class proportions.
result Improves balanced accuracy and sensitivity for minority class (toxic compounds).
A new method corrects weight values to improve treatment effect estimation.
problem Estimating heterogeneous treatment effects in high-dimensional data with sample selection bias.
method Differentiable Pareto-Smoothed Weighting (DPSW) framework.
result Our method outperforms existing methods in treatment effect estimation.
Model for material elasticity and plasticity using networks.
problem Understanding the elasticity and plasticity of materials.
method Developed a mathematical model based on networks, defining tension tensor for periodic graphs.
result The model explains elasticity and plasticity through local moves on graphs.
Not all instances in a data set are equally beneficial for inducing a model of the data. Some instances (such as outliers or noise) can be detrimental. However, at least initially, the instances in a data set are generally considered equally in machine learning algorithms. Many current approaches for handling noisy and…
This paper investigates weight-sharing in NAS, revealing its impact and providing solutions.
problem Reducing the time and computational cost of training neural networks.
method Comprehensive experiments on weight-sharing in NAS, analyzing variance and interference.
result Properly reducing weight sharing can reduce variance and improve model performance.
Koopman mode analysis applied to neural networks for training optimization.
problem Optimizing neural network training, identifying issues, and speeding up learning.
method Koopman operator analysis of neural network dynamics.
result Spectral analysis of Koopman operator aids in determining network depth, initialization quality, and training termination.
In importance sampling (IS)-based reinforcement learning algorithms such as Proximal Policy Optimization (PPO), IS weights are typically clipped to avoid large variance in learning. However, policy update from clipped statistics induces large bias in tasks with high action dimensions, and bias from clipping makes it di…
Optimal discrete harmonic maps between hyperbolic surfaces are found via minimizing energy.
problem Finding optimal discrete harmonic maps between hyperbolic surfaces.
method Minimizing Dirichlet energy over all possible hyperbolic structures and realizations within a fixed homotopy class.
result At the optimal hyperbolic structure, the discrete harmonic map and edge weights are induced from a weighted Delaunay decomposition.
We prove several results about chordal graphs and weighted chordal graphs by focusing on exposed edges. These are edges that are properly contained in a single maximal complete subgraph. This leads to a characterization of chordal graphs via deletions of a sequence of exposed edges from a complete graph. Most interesti…
TILT improves target domain performance by penalizing an auxiliary component on unlabeled target inputs.
problem Improving performance on target domain under covariate shift.
method TILT uses a novel objective function to decompose the source predictor and penalize an auxiliary component on unlabeled target inputs.
result TILT improves target domain performance over source-only training and other baselines.
Optimizes pruning masks for neural networks using probabilistic fine-tuning and PAC-Bayes bounds.
problem Improving neural network performance through adaptive pruning of weights.
method Optimizes stochastic pruning masks by minimizing expected loss, considering data-adaptive regularization and feature alignment.
result Probabilistic fine-tuning leads to improved test error over baseline methods in neural networks.
Inverse classification uses an induced classifier as a queryable oracle to guide test instances towards a preferred posterior class label. The result produced from the process is a set of instance-specific feature perturbations, or recommendations, that optimally improve the probability of the class label. In this work…
STR reparameterizes DNN weights with soft thresholds for better sparsity and accuracy.
problem Improving sparsity in DNNs for better accuracy and lower inference cost.
method Soft Threshold Reparameterization (STR) using the soft-threshold operator on DNN weights.
result STR achieves state-of-the-art accuracy and reduces FLOPs by up to 50%.
Bayesian neural networks can be simplified by parameterizing weights as rank-r matrices, reducing parameter count and improving performance.
problem High parameter count in standard Bayesian neural networks.
method Parameterize weights as W=ABop with A∈Rmimesr, B∈Rnimesr, inducing a singular posterior. result PAC-Bayes generalization bounds and loss bounds show improved performance with fewer parameters.
We consider a setting where an agent's uncertainty is represented by a set of probability measures, rather than a single measure. Measure-bymeasure updating of such a set of measures upon acquiring new information is well-known to suffer from problems; agents are not always able to learn appropriately. To deal with the…
Kähler information manifolds for signal filters in weighted Hardy spaces are explored.
problem Developing a geometric framework for signal processing filters in weighted Hardy spaces.
method Introducing weighted Hardy spaces and smooth transformations of transfer functions, demonstrating the Kähler manifold structure.
result The Riemannian geometry of weighted Hardy norms for transfer functions forms a Kähler manifold.