Theory explains neural network scaling with dataset and model size.
problem Neural network scaling laws with dataset and model size.
method Identified variance-limited and resolution-limited scaling behaviors.
result Four scaling regimes explained: infinite data, infinite width, resolution-limited, and large width.
Paper introduces scalable clustering for large datasets with outliers.
problem Lack of scalable algorithms for large datasets with outliers.
method Provable robust clustering algorithm based on loss minimization for Gaussian mixture models.
result Algorithm provides high accuracy with theoretical guarantees and outperforms existing methods.
This work aims to create a large-scale model for critical care time series data.
problem Lack of large-scale datasets and distribution shifts in critical care time series data.
method Harmonized dataset creation and transfer learning research.
result Established a foundation for large-scale multi-variate time series models in critical care.
New kernel model scales to large datasets.
problem Challenges in scaling kernel machines to large datasets.
method Introduced EigenPro 3.0, an algorithm based on projected dual preconditioned SGD.
result Ability to scale model and data sizes independently.
New method speeds up Gaussian process training and inference for large datasets.
problem Training and inference in Gaussian processes are computationally expensive for large datasets.
method Iterative alternating projection method that accesses subblocks of the kernel matrix, reducing time and space complexity.
result Empirically, the method accelerates GP training and inference by up to 72x compared to conjugate gradients.
This paper compresses large datasets for efficient machine learning.
problem Efficiently processing large datasets for machine learning.
method Constructing a sketch of the dataset using random features and averaging, then learning from the sketch.
result The approach can perform machine learning tasks without full dataset access, preserving both information and privacy.
Large models follow power laws in performance with dataset size or parameters.
problem Understanding neural scaling laws in large language models.
method Joint generative data model and random feature model.
result Modeling and solving the dual limit reveals insights into scaling laws.
The paper proposes an efficient method to scale Bayesian inference for mixed multinomial logit models to very large datasets.
problem Efficiency in Bayesian inference for mixed multinomial logit models on large datasets.
method Amortized Variational Inference with stochastic backpropagation, automatic differentiation, and GPU acceleration.
result The proposed method achieves significant computational speedups over traditional methods for large datasets.
Paper proposes data quality measures for large-scale high-dimensional data.
problem Lack of practical data quality measures for large-scale high-dimensional data.
method Proposes two data quality measures: class separability and in-class variability. Efficient algorithms based on random projections and bootstrapping are provided.
result Efficient algorithms for computing data quality measures on large-scale high-dimensional data.
This paper introduces a new dataset called "ToyADMOS" designed for anomaly detection in machine operating sounds (ADMOS). To the best our knowledge, no large-scale datasets are available for ADMOS, although large-scale datasets have contributed to recent advancements in acoustic signal processing. This is because anoma…
This work improves Gaussian process model selection for large datasets.
problem Prohibitively high computational cost in Gaussian process model selection.
method Linear-time scaling and computational uncertainty tradeoff.
result Computation-aware Gaussian processes can be trained on large datasets efficiently.
Typically, Softmax is used in the final layer of a neural network to get a probability distribution for output classes. But the main problem with Softmax is that it is computationally expensive for large scale data sets with large number of possible outputs. To approximate class probability efficiently on such large sc…
Gradient-free deep learning for large datasets.
problem Training deep neural networks on large-scale datasets is resource-intensive and requires specialized techniques.
method Recursive Local Representation Alignment (RLRA) for gradient-free training.
result RLRA achieves comparable performance to backprop while converging faster and being parallelizable.
ParK efficiently solves kernel ridge regression for large datasets.
problem Large-scale kernel ridge regression efficiency and accuracy.
method Partitioning feature space with random projections and iterative optimization.
result Provably maintains statistical accuracy with reduced space and time complexity.
Quantum machine learning tackles large datasets with randomized measurements.
problem Efficiently process large, high-dimensional datasets on quantum computers.
method Randomized measurements to scale linearly with dataset size and quadratic for post-processing.
result Substantial speed-up for noisy quantum computers, enabling image classification.
AcceleratedLiNGAM speeds up causal discovery methods for large datasets.
problem Slow causal discovery methods for large-scale datasets.
method Parallelized LiNGAM method with GPU acceleration.
result Up to 32-fold speed-up on benchmark datasets.
The sample inefficiency of standard deep reinforcement learning methods precludes their application to many real-world problems. Methods which leverage human demonstrations require fewer samples but have been researched less. As demonstrated in the computer vision and natural language processing communities, large-scal…
A method for large scale Gaussian process classification has been recently proposed based on expectation propagation (EP). Such a method allows Gaussian process classifiers to be trained on very large datasets that were out of the reach of previous deployments of EP and has been shown to be competitive with related tec…
Kernel clustering algorithm improved for large datasets using incomplete Cholesky factorization.
problem Large memory usage in kernel-based clustering for large-scale datasets.
method Approximate the kernel matrix using incomplete Cholesky factorization and apply linear k-means clustering. result The proposed method achieves similar performance to kernel k-means clustering but handles large-scale datasets efficiently. Two large medical dialogue datasets for improving healthcare.
problem Improving healthcare through better dialogue systems.
method Building two large-scale medical dialogue datasets: MedDialog-EN and MedDialog-CN.
result The datasets are the largest medical dialogue datasets to date.
Large-scale Hierarchical Classification (HC) involves datasets consisting of thousands of classes and millions of training instances with high-dimensional features posing several big data challenges. Feature selection that aims to select the subset of discriminant features is an effective strategy to deal with large-sc…
FibeRed reduces complex data dimensions while preserving topology.
problem Hard embedding of topologically complex datasets in low-dimensional Euclidean space.
method Modeling datasets with vector bundles, reducing fibers while preserving topology.
result FibeRed learns topologically faithful embeddings in lower dimensions than existing methods.
ADASAP accelerates GP inference for large datasets.
problem Scaling Gaussian process inference to large datasets.
method Approximate sketch-and-project algorithm for solving linear systems.
result Sketch-and-project rapidly converges to true posterior mean.
We study large-scale classification problems in changing environments where a small part of the dataset is modified, and the effect of the data modification must be quickly incorporated into the classifier. When the entire dataset is large, even if the amount of the data modification is fairly small, the computational …
Efficiently applies NTK to large-scale datasets using random features.
problem Computational limitations of kernel methods for large-scale datasets.
method Proposes a sketching-based algorithm combining random features of arc-cosine kernels to construct an efficient feature map of the NTK.
result Achieves comparable error bounds to exact kernel methods but with significantly reduced feature dimensionality.
Video applications and analytics are routinely projected as a stressing and significant service of the Nationwide Public Safety Broadband Network. As part of a NIST PSCR funded effort, the New Jersey Office of Homeland Security and Preparedness and MIT Lincoln Laboratory have been developing a computer vision dataset o…
Feature selection is an important challenge in machine learning. It plays a crucial role in the explainability of machine-driven decisions that are rapidly permeating throughout modern society. Unfortunately, the explosion in the size and dimensionality of real-world datasets poses a severe challenge to standard featur…
A new large-scale tabular benchmark for Learning from Label Proportions.
problem Lack of a large-scale open benchmark for tabular Learning from Label Proportions.
method Proposed LLP-Bench, a suite of 70 datasets (62 feature bag and 8 random bag) from real-world tabular data.
result Demonstrated the effectiveness of 9 SOTA and popular tabular LLP techniques on 62 feature bag datasets.
Kernel methods are powerful and flexible approach to solve many problems in machine learning. Due to the pairwise evaluations in kernel methods, the complexity of kernel computation grows as the data size increases; thus the applicability of kernel methods is limited for large scale datasets. Random Fourier Features (R…
New method speeds up training of large kernel models.
problem Scaling kernel machines to large datasets and model sizes.
method Delayed projections in Preconditioned Stochastic Gradient Descent (PSGD).
result Significant training speed up over existing methods.
SEMASIA provides a large dataset of latent representations for model comparison.
problem Difficulty in comparing semantic structures across different neural network models.
method Collection of latent representations from 1700 pretrained models across various benchmarks.
result Consistent semantic organization across models and datasets.
Bilinear models such as DistMult and ComplEx are effective methods for knowledge graph (KG) completion. However, they require large batch sizes, which becomes a performance bottleneck when training on large scale datasets due to memory constraints. In this paper we use occurrences of entity-relation pairs in the datase…
Deep Neural Networks (DNNs) often rely on very large datasets for training. Given the large size of such datasets, it is conceivable that they contain certain samples that either do not contribute or negatively impact the DNN's optimization. Modifying the training distribution in a way that excludes such samples could …
Optimal scaling found to depend on operator norm across large models and datasets.
problem Lack of unifying principle for optimal hyperparameter scaling across models and datasets.
method Discovered that optimal scaling is conditioned on the operator norm of the output layer.
result The optimal learning rate/batch size pair (η∗,B∗) consistently has the same operator norm value. We propose a new method for input variable selection in nonlinear regression. The method is embedded into a kernel regression machine that can model general nonlinear functions, not being a priori limited to additive models. This is the first kernel-based variable selection method applicable to large datasets. It sides…
New scalable methods for robust model learning from large datasets.
problem Training robust models resistant to data distribution shifts.
method Composite optimization for distributionally robust optimization (DRO).
result Scalable methods for learning robust models from large datasets.
Deep neural networks (DNNs) trained on large-scale datasets have exhibited significant performance in image classification. Many large-scale datasets are collected from websites, however they tend to contain inaccurate labels that are termed as noisy labels. Training on such noisy labeled datasets causes performance de…
Researchers parallelize neural kernels for large-scale data, achieving state-of-the-art accuracy.
problem Limited scalability of neural kernels on large datasets.
method Massively parallel computation across many GPUs, combined with a distributed, preconditioned conjugate gradients algorithm.
result Achieved state-of-the-art accuracy of 91.2% on CIFAR-5m dataset using neural kernels.
Paper presents an efficient algorithm for learning minimax risk classifiers with large-scale data.
problem Efficient learning of minimax risk classifiers for large-scale data with multiple classes.
method Combination of constraint and column generation for efficient learning.
result 10x speedup for general large-scale data and 100x speedup with many classes.
This work extends the scaling law to multiple and kernel regression, challenging traditional machine learning principles.
problem Challenging traditional machine learning wisdom with scaling law in large practical models.
method Demonstrates the scaling law in multiple and kernel regression settings.
result The scaling law extends to multiple and kernel regression, providing deeper insights into LLMs.
The paper shows how to efficiently generate large Gaussian process samples with reliability guarantees.
problem Generating large-scale Gaussian process samples efficiently and with reliability.
method Demonstrates scaling data generation to large \(n\) while providing high probability guarantees.
result Efficiently generates large Gaussian process samples with reliability guarantees.
Many real-world regression problems demand a measure of the uncertainty associated with each prediction. Standard decision forests deliver efficient state-of-the-art predictive performance, but high-quality uncertainty estimates are lacking. Gaussian processes (GPs) deliver uncertainty estimates, but scaling GPs to lar…
This paper proposes a new method for learning covers of geometric datasets to improve topological inference and visualization.
problem Improving topological inference and visualization of large-scale geometric datasets.
method Proposes a method for learning topologically-faithful covers of geometric datasets using optimization.
result Simplicial complexes obtained from learned covers outperform standard methods in terms of size and representation of large-scale topology.
Recent work has shown how to train Convolutional Neural Networks (CNNs) rapidly on large image datasets, then transfer the knowledge gained from these models to a variety of tasks. Following [Radford 2017], in this work, we demonstrate similar scalability and transfer for Recurrent Neural Networks (RNNs) for Natural La…
Supervised learning algorithms are nowadays successfully scaling up to datasets that are very large in volume, leveraging the potential of in-memory cluster-computing Big Data frameworks. Still, massive datasets with a number of large-domain categorical features are a difficult challenge for any classifier. Most off-th…
Scalable kernel methods for large datasets using Fourier representations and NUFFT.
problem Cubic complexity in kernel methods limits their use on large-scale datasets.
method Fourier representation of kernels combined with NUFFT for O(n log n) complexity.
result Achieves minimax convergence rates and processes up to tens of billions of samples.
Multi-label classification has received considerable interest in recent years. Multi-label classifiers have to address many problems including: handling large-scale datasets with many instances and a large set of labels, compensating missing label assignments in the training set, considering correlations between labels…
CHILI datasets tackle inorganic nanomaterials, advancing graph machine learning.
problem Challenges in modelling inorganic crystalline materials and nanomaterials with graph ML.
method Presented two large-scale datasets of inorganic nanomaterials, defined property and structure prediction tasks.
result Benchmarked performance of graph ML methods on inorganic nanomaterials, highlighting areas for future work.