A new dataset distance using optimal transport, agnostic of model and label sets.
problem Quantifying task similarity across datasets without model dependence.
method Optimal transport for model-agnostic dataset comparison.
result The new distance correlates with transfer learning difficulty across various datasets.
Two novel dataset optimization strategies improve malware traffic detection accuracy.
problem Redundant and irrelevant information in network traffic datasets increases computational cost and noise.
method Feature selection and dimensional reduction techniques using mutual information and autoencoders.
result Optimized dataset leads to improved accuracy of Multi Layer Perceptron for malware detection.
Optimal quantization improves dataset distillation for faster training.
problem Efficiently train models with synthetic data.
method Reformulate disentangled methods as optimal quantization problems.
result Better performance and generalization on ImageNet-1K and subsets.
Hyperparameter optimization aims to find the optimal hyperparameter configuration of a machine learning model, which provides the best performance on a validation dataset. Manual search usually leads to get stuck in a local hyperparameter configuration, and heavily depends on human intuition and experience. A simple al…
s-OTDD compares datasets efficiently without training, robust to class variations.
problem Efficiently compare datasets without training or class variations.
method Moment Transform Projection (MTP) and sliced optimal transport.
result s-OTDD correlates with optimal transport and transfer learning performance.
Meta-learning symbolic default hyperparameters from dataset properties.
problem Empirical hyperparameter optimization is slow and requires manual configuration.
method Evolutionary algorithm to learn symbolic hyperparameter formulas from dataset properties.
result Meta-learning finds viable symbolic defaults for ML algorithms.
A new method distills datasets more efficiently and effectively.
problem Achieving competitive performance on test data with a small synthetic dataset.
method Tackles dataset distillation as a bilevel optimization problem, introduces RaT-BPTT to stabilize gradients and speed up optimization.
result Establishes new state-of-the-art performance across various benchmarks.
Paper uses Bayesian optimization to find best Supertrend indicator settings.
problem Finding optimal trading parameters for the Supertrend indicator.
method Bayesian optimization to automate parameter selection.
result BO-optimized Supertrend strategy yields higher profits in backtesting.
Optimal scaling found to depend on operator norm across large models and datasets.
problem Lack of unifying principle for optimal hyperparameter scaling across models and datasets.
method Discovered that optimal scaling is conditioned on the operator norm of the output layer.
result The optimal learning rate/batch size pair (η∗,B∗) consistently has the same operator norm value. New scalable methods for robust model learning from large datasets.
problem Training robust models resistant to data distribution shifts.
method Composite optimization for distributionally robust optimization (DRO).
result Scalable methods for learning robust models from large datasets.
Bayesian optimization has become a successful tool for hyperparameter optimization of machine learning algorithms, such as support vector machines or deep neural networks. Despite its success, for large datasets, training and validating a single configuration often takes hours, days, or even weeks, which limits the ach…
Bispectral OT improves dataset comparison by preserving intrinsic coherence.
problem Ignoring intrinsic coherence in dataset comparisons using pairwise geometric distances.
method Introduces Bispectral Optimal Transport, a symmetry-aware extension of discrete OT.
result Transport plans computed with Bispectral OT achieve greater class preservation accuracy.
We provide the construction of a set of square matrices whose translates and rotates provide a Parseval frame that is optimal for approximating a given dataset of images. Our approach is based on abstract harmonic analysis techniques. Optimality is considered with respect to the quadratic error of approximation of the …
Develops efficient algorithm for distilling supervised and offline RL datasets.
problem Creating synthetic datasets for supervised and offline RL models to match training data performance.
method An efficient algorithm based on matching losses without model training, using randomly sampled regressors.
result Synthetic datasets derived with ildeO(d2) sampled regressors match MSE loss of bounded linear models on training data. The success of lottery ticket initializations (Frankle and Carbin, 2019) suggests that small, sparsified networks can be trained so long as the network is initialized appropriately. Unfortunately, finding these "winning ticket" initializations is computationally expensive. One potential solution is to reuse the same wi…
DGKIP extends KIP for dataset distillation without bi-level optimization.
problem Efficiently distill datasets for various loss functions.
method Leverages duality theory to avoid bi-level optimization.
result DGKIP supports a wider range of loss functions.
SPlit optimizes dataset splitting for better model performance.
problem Improving model performance through optimal dataset splitting.
method Adapting Support Points (SP) algorithm for subsampling and categorical variables in a sequential nearest neighbor approach.
result SPlit significantly improves worst-case testing performance compared to random splitting.
When hyperparameter optimization of a machine learning algorithm is repeated for multiple datasets it is possible to transfer knowledge to an optimization run on a new dataset. We develop a new hyperparameter-free ensemble model for Bayesian optimization that is a generalization of two existing transfer learning extens…
We consider the problem of publicly releasing a dataset for support vector machine classification while not infringing on the privacy of data subjects (i.e., individuals whose private information is stored in the dataset). The dataset is systematically obfuscated using an additive noise for privacy protection. Motivate…
Automated machine learning aims to automate the whole process of machine learning, including model configuration. In this paper, we focus on automated hyperparameter optimization (HPO) based on sequential model-based optimization (SMBO). Though conventional SMBO algorithms work well when abundant HPO trials are availab…
TaskSet dataset speeds up hyperparameter optimization.
problem Optimizing hyperparameters for various machine learning tasks.
method Meta-learning an ordered list of hyperparameters from a large dataset.
result Meta-learning achieves large speedups in sample efficiency.
First private Bayesian optimization algorithm with provable performance.
problem Privacy-preserving Bayesian optimization in outsourced settings.
method Random projection-based transformation to preserve privacy and distances.
result Regret bound similar to non-private GP-UCB established for PO-GP-UCB.
A new method for aligning datasets without known correspondences.
problem Aligning datasets from different domains without labeled correspondences.
method Integrates MDS and Wasserstein Procrustes for joint optimization of embeddings and correspondences.
result Maps datasets to a common low-dimensional space without labeled correspondences.
Partial Wasserstein Covering aims to identify missing patterns in datasets.
problem Identifying missing patterns in datasets compared to actual applications.
method Formulated as a discrete optimization problem with partial Wasserstein divergence. Proved submodular, allowing greedy approximation. Proposed quasi-greedy algorithms with acceleration techniques.
result Efficiently fills gaps and finds missing scenes in real driving scenes datasets.
Labels distilled from images improve model training efficiency and flexibility.
problem Creating synthetic labels for a small set of real images to train models effectively.
method Introduce a more robust and flexible meta-learning algorithm for distillation and an effective first-order strategy based on convex optimization layers.
result Label distillation leads to improved results and greater flexibility in neural architectures.
Deep neural networks (DNNs) trained on large-scale datasets have exhibited significant performance in image classification. Many large-scale datasets are collected from websites, however they tend to contain inaccurate labels that are termed as noisy labels. Training on such noisy labeled datasets causes performance de…
PASTA optimizes assortment selection using pessimism principle.
problem Optimizing assortment selection with limited data coverage.
method Pessimistic Assortment Optimization (PASTA) based on the principle of pessimism.
result PASTA correctly identifies optimal assortment with minimal data coverage.
New financial dataset and model detect claims affecting market returns.
problem Detecting analyst claims' impact on financial markets.
method Constructed new dataset, used weak-supervision model with SME knowledge.
result Outperformed existing models in claim detection and market analysis.
This work attempts to find the most optimal parameter setting of a deep artificial neural network (ANN) for Bengali digit dataset by pre-training it using stacked denoising autoencoder (SDA). Although SDA based recognition is hugely popular in image, speech and language processing related tasks among the researchers, i…
Sketching algorithm finds closest point on convex hull efficiently.
problem Finding the closest point on a convex hull of large datasets.
method Sketching procedure to exploit data structure, gradient project method.
result Faster solution than standard optimization algorithms.
Inferring a decision tree from a given dataset is one of the classic problems in machine learning. This problem consists of buildings, from a labelled dataset, a tree such that each node corresponds to a class and a path between the tree root and a leaf corresponds to a conjunction of features to be satisfied in this c…
Sparse Convex Biclustering improves accuracy and robustness in high-dimensional datasets.
problem Challenges in clustering rows and columns of large-scale datasets due to noise and computational complexity.
method Sparse Convex Biclustering (SpaCoBi) using convex optimization and stability-based tuning.
result Significantly outperforms state-of-the-art methods in accuracy for high-dimensional datasets.
Optimal Gaussian noise mechanisms achieve nearly optimal error in unbiased mean estimation.
problem Efficiently estimating the mean of high-dimensional data while preserving privacy.
method Differential privacy mechanisms with Gaussian noise, focusing on optimal covariance.
result Gaussian noise mechanisms achieve nearly optimal error among all private unbiased mean estimation mechanisms.
PropEn uses matching to create a larger dataset for efficient design optimization.
problem Limited data and complex landscapes in scientific applications.
method PropEn uses a matching approach to implicitly guide design without a discriminator.
result PropEn efficiently approximates the gradient of property improvement within the data distribution.
FanG-HPO optimizes machine learning models for fairness and low energy consumption.
problem Bias in machine learning models and high energy consumption in hyperparameter optimization.
method Combines multi-objective and multiple information source Bayesian optimization.
result FanG-HPO identifies fair and energy-efficient machine learning models.
Data whitening and second order optimization harm generalization by reducing access to dataset information.
problem Harmful effects of data whitening and second order optimization on generalization in machine learning.
method Analysis of fully connected models and experimental verification.
result Data whitening and second order optimization reduce or prevent generalization by limiting access to dataset information.
PASHA optimizes model tuning for large datasets with limited resources.
problem Expensive HPO and NAS for large datasets.
method Dynamic resource allocation approach.
result Significantly reduces computational resources while maintaining performance.
Simultaneous clustering and optimization (SCO) has recently drawn much attention due to its wide range of practical applications. Many methods have been previously proposed to solve this problem and obtain the optimal model. However, when a dataset evolves over time, those existing methods have to update the model freq…
Paper optimizes GAIL for online and offline learning with linear approximations.
problem Imitation learning from expert demonstrations with linear function approximations.
method Proposes optimistic and pessimistic algorithms for online and offline settings.
result Proves optimality and efficiency of proposed algorithms.
Optimal transport distances help impute missing data.
problem Missing data in real-world datasets.
method Use optimal transport distances as a loss function to impute missing data values.
result OT-based methods match or outperform state-of-the-art imputation methods.
Most users of online services have unique behavioral or usage patterns. These behavioral patterns can be exploited to identify and track users by using only the observed patterns in the behavior. We study the task of identifying users from statistics of their behavioral patterns. Specifically, we focus on the setting i…
Recently, several optimization methods have been successfully applied to the hyperparameter optimization of deep neural networks (DNNs). The methods work by modeling the joint distribution of hyperparameter values and corresponding error. Those methods become less practical when applied to modern DNNs whose training ma…
This work improves Gaussian process model selection for large datasets.
problem Prohibitively high computational cost in Gaussian process model selection.
method Linear-time scaling and computational uncertainty tradeoff.
result Computation-aware Gaussian processes can be trained on large datasets efficiently.
A new offline RL framework unifies imitation learning and vanilla offline RL.
problem Learning from expert datasets without active data collection.
method A new offline RL framework that interpolates between imitation learning and vanilla offline RL, using a weak concentrability coefficient and a lower confidence bound algorithm.
result LCB algorithm achieves a faster rate of 1/N for nearly-expert datasets, and is adaptively optimal for the entire data composition range. A new method reduces energy consumption in machine learning by using multiple, less costly data sources.
problem High computational and energy costs in machine learning model training.
method Augmented Gaussian Process (AGP-MISO) with multi-source optimization.
result The AGP-MISO method reduces computational time and energy consumption compared to traditional approaches.
Combinatorial optimization problems for clustering are known to be NP-hard. Most optimization methods are not able to find the global optimum solution for all datasets. To solve this problem, we propose a global optimal path-based clustering (GOPC) algorithm in this paper. The GOPC algorithm is based on two facts: (1) …
Study on SignGD optimization of two-layer transformer on noisy data.
problem Understanding how SignGD optimizes transformers and its generalization.
method Analysis of a two-layer transformer with SignGD on a linearly separable noisy dataset.
result SignGD converges fast but has poor generalization on noisy data.
This work studies the impact of intra-/inter-class diversity on pre-training datasets and finds a balance for optimal performance.
problem The impact of intra-/inter-class diversity on supervised pre-training datasets and their effect on downstream tasks.
method Empirical study and theoretical analysis of the relationship between diversity types and downstream performance.
result The optimal class-to-sample ratio is invariant to the size of the pre-training dataset and can be predicted.