The performance of classification algorithms with a massive and highly imbalanced data stream depends upon efficient balancing strategy. Some techniques of balancing strategy have been applied in the past with Batch data to resolve the class imbalance problem. This paper proposes a new incremental data balancing framew…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “massive data”
Efficiently bootstraps massive distributed data without over-resampling.
problem Statistical inference for massive distributed data.
method Distributed Bootstrap applied to gradients from worker machines.
result Proves optimal statistical efficiency with minimal communication.
New algorithms for fair data summarization in massive data models.
problem Fair -center problem in massive data.
method Streaming and distributed algorithms with provable guarantees.
result First distributed algorithm with constant approximation ratio.
Develops a new theory for approximating functions on massive data.
problem Challenges in machine learning with massive data.
method eignets theory for local, stratified approximation.
result Solves inverse problems like finding data probability law and function smoothness.
The great success of deep learning poses urgent challenges for understanding its working mechanism and rationality. The depth, structure, and massive size of the data are recognized to be three key ingredients for deep learning. Most of the recent theoretical studies for deep learning focus on the necessity and advanta…
New method for efficient inference in large datasets.
problem Statistical inference in massive datasets.
method Combines divide-and-conquer method and empirical likelihood.
result Reduces computation burden and demonstrates effectiveness.
We present a Bayesian non-negative tensor factorization model for count-valued tensor data, and develop scalable inference algorithms (both batch and online) for dealing with massive tensors. Our generative model can handle overdispersed counts as well as infer the rank of the decomposition. Moreover, leveraging a repa…
This paper provides an initial investigation on the application of convolutional neural networks (CNNs) for fingerprint-based positioning using measured massive MIMO channels. When represented in appropriate domains, massive MIMO channels have a sparse structure which can be efficiently learned by CNNs for positioning …
Massively multitask neural architectures provide a learning framework for drug discovery that synthesizes information from many distinct biological sources. To train these architectures at scale, we gather large amounts of data from public sources to create a dataset of nearly 40 million measurements across more than 2…
ADDA framework speeds up data augmentation in massive data settings.
problem Slow data augmentation in massive data settings.
method Develops asynchronous and distributed data augmentation (ADDA) framework.
result ADDA significantly speeds up data augmentation compared to parent DA algorithms.
SDCOR clusters massive datasets efficiently, detecting outliers with low memory usage.
problem Local outlier detection in large-scale datasets.
method Chunk-based density clustering with incremental updates.
result SDCOR achieves lower linear time complexity and better efficiency than traditional methods.
We present a scalable Bayesian model for low-rank factorization of massive tensors with binary observations. The proposed model has the following key properties: (1) in contrast to the models based on the logistic or probit likelihood, using a zero-truncated Poisson likelihood for binary data allows our model to scale …
Deep learning reduces training overhead in massive MIMO systems.
problem Reducing training overhead in massive MIMO systems.
method Use of deep learning (NNs) to improve CSI acquisition and feedback processes.
result Significant improvements in performance and reduced complexity.
Massive fermions help understand index theorems without chiral symmetry.
problem Understanding index theorems in massive fermion systems.
method Reformulate chiral anomaly and index theorems with massive Dirac operators.
result Nontrivial mathematical relations between massless and massive fermions.
Improved spatial prediction for massive datasets using SME model.
problem Efficiently estimating parameters in massive spatial datasets.
method Spatial Mixed Effects (SME) model with AECM algorithm for flexibility.
result Improved estimation without sacrificing prediction accuracy.
Data collection at a massive scale is becoming ubiquitous in a wide variety of settings, from vast offline databases to streaming real-time information. Learning algorithms deployed in such contexts must rely on single-pass inference, where the data history is never revisited. In streaming contexts, learning must also …
Variables in many massive high-dimensional data sets are structured, arising for example from measurements on a regular grid as in imaging and time series or from spatial-temporal measurements as in climate studies. Classical multivariate techniques ignore these structural relationships often resulting in poor performa…
Develops an online nonparametric classifier for massive data.
problem Challenges of batch kernel-based nonparametric classifiers in massive data.
method Online principle components analysis to reduce dimensionality, followed by stochastic approximation algorithm for real-time calculation.
result Online classifier provides the best trade-off between accuracy and computation cost.
Random orthogonalization improves FL in massive MIMO systems without CSI.
problem Efficient model aggregation in FL with minimal channel estimation overhead.
method Combining FL with massive MIMO's channel hardening and favorable propagation, random orthogonalization reduces channel estimation overhead.
result Achieves model aggregation without CSI, significantly reducing channel estimation overhead.
Data analysis and data mining are concerned with unsupervised pattern finding and structure determination in data sets. "Structure" can be understood as symmetry and a range of symmetries are expressed by hierarchy. Such symmetries directly point to invariants, that pinpoint intrinsic properties of the data and of the …
Paper tackles missing data in medical records using advanced optimization.
problem Missing data in medical records hinders machine learning applications.
method Applied multi-level stochastic optimization for imputation.
result The method significantly outperforms current approaches and reduces error up to 75%.
Cluster analysis of very high dimensional data can benefit from the properties of such high dimensionality. Informally expressed, in this work, our focus is on the analogous situation when the dimensionality is moderate to small, relative to a massively sized set of observations. Mathematically expressed, these are the…
Sparse matrix factorization is a popular tool to obtain interpretable data decompositions, which are also effective to perform data completion or denoising. Its applicability to large datasets has been addressed with online and randomized methods, that reduce the complexity in one of the matrix dimension, but not in bo…
COMET is a single-pass MapReduce algorithm for learning on large-scale data. It builds multiple random forest ensembles on distributed blocks of data and merges them into a mega-ensemble. This approach is appropriate when learning from massive-scale data that is too large to fit on a single machine. To get the best acc…
Study evaluates topological contributions in massive SQCD on compact 4-manifolds.
problem Analyzing topological path integrals for massive SQCD with up to 3 massive hypermultiplets.
method Decouples hypermultiplets, evaluates massless limit, and merges singularities at Argyres-Douglas points. Uses mass expansions for and .
result Physical partition functions match mathematical results on Segre numbers of instanton moduli spaces.
Massively multi-label prediction/classification problems arise in environments like health-care or biology where very precise predictions are useful. One challenge with massively multi-label problems is that there is often a long-tailed frequency distribution for the labels, which results in few positive examples for t…
Unified framework for photon and massive particle hypersurfaces in stationary spacetimes.
problem Understanding photon and massive particle hypersurfaces in stationary spacetimes.
method Unified framework using Killing-invariant timelike hypersurfaces and associated Finsler structures.
result Conditions for a hypersurface to be a photon or massive particle hypersurface are established.
This paper investigates the grant-free random access with massive IoT devices. By embedding the data symbols in the signature sequences, joint device activity detection and data decoding can be achieved, which, however, significantly increases the computational complexity. Coordinate descent algorithms that enjoy a low…
In this work we simulate null geodesics for the Bonnor massive dipole metric by implementing a symbolic-numerical algorithm in Sage and Python. This program is also capable of visualizing in 3D, in principle, the geodesics for any given metric. Geodesics are launched from a common point, collectively forming a cone of …
Topic models are popular for modeling discrete data (e.g., texts, images, videos, links), and provide an efficient way to discover hidden structures/semantics in massive data. One of the core problems in this field is the posterior inference for individual data instances. This problem is particularly important in strea…
Introduces a massive variant of Ray-Singer Torsion to avoid zero modes in topological field theories.
problem Avoiding zero modes in the evaluation of path integrals for topological field theories.
method Introduces a massive variant of the Ray-Singer Torsion, involving determinants of the twisted Laplacian with mass but without zero modes.
result Explicitly evaluates the massive Ray-Singer Torsion on product manifolds and mapping tori.
The Sinkhorn "distance", a variant of the Wasserstein distance with entropic regularization, is an increasingly popular tool in machine learning and statistical inference. However, the time and memory requirements of standard algorithms for computing this distance grow quadratically with the size of the data, making th…
QEM uses parallel importance weighting for fast approximate Bayesian inference.
problem Bayesian inference challenges in large models with many observations and latent variables.
method Expectation Maximization (EM) with massively parallel importance weighting.
result QEM is faster and more scalable than RWS and VI.
Paper proposes a GPU-based system for training massive deep learning models in ads systems.
problem Training massive deep learning models with terabyte-scale parameters in ads systems.
method Hierarchical GPU parameter server with 3-layer storage (GPU High-Bandwidth Memory, CPU main memory, SSD).
result 4-node hierarchical GPU parameter server trains a model 2X faster than a 150-node in-memory system.
Opioid addiction is a severe public health threat in the U.S, causing massive deaths and many social problems. Accurate relapse prediction is of practical importance for recovering patients since relapse prediction promotes timely relapse preventions that help patients stay clean. In this paper, we introduce a Generati…
The in-game economies of massively multi-player online games (MMOGs) are complex systems that have to be carefully designed and managed. This paper presents the results of an analysis of auction house data from the MMOG Glitch, across a 14 month time period, the entire lifetime of the game. The data comprise almost 3 m…
In the era of big data, it is desired to develop efficient machine learning algorithms to tackle massive data challenges such as storage bottleneck, algorithmic scalability, and interpretability. In this paper, we develop a novel efficient classification algorithm, called fast polynomial kernel classification (FPC), to…
In recent years, a rich variety of shrinkage priors have been proposed that have great promise in addressing massive regression problems. In general, these new priors can be expressed as scale mixtures of normals, but have more complex forms and better properties than traditional Cauchy and double exponential priors. W…
Study improves scalability of cell-free massive MIMO networks by optimizing UE-AP association.
problem Optimizing UE-AP association in cell-free massive MIMO networks.
method Deep learning algorithm using Bidirectional Long Short-Term Memory cells and hybrid probabilistic weight updating.
result Enhanced scalability without retraining, robust against pilot contamination.
A fundamental aspect of biological information processing is the ubiquity of sequence-function relationships -- functions that map the sequence of DNA, RNA, or protein to a biochemically relevant activity. Most sequence-function relationships in biology are quantitative, but only recently have experimental techniques f…
The rapid growth of Electronic Health Records (EHRs), as well as the accompanied opportunities in Data-Driven Healthcare (DDH), has been attracting widespread interests and attentions. Recent progress in the design and applications of deep learning methods has shown promising results and is forcing massive changes in h…
Neural MMO v1.3: A Massively Multiagent Game Environment for Training and Evaluating Neural Networkscs.LG
Neural MMO simulates MMOs to study multiagent intelligence.
problem Limited research environments for multiagent intelligence.
method Developed a new game environment inspired by MMOs.
result Standard methods can learn interesting behaviors in MMOs.
DFRot: Achieving Outlier-Free and Massive Activation-Free for Rotated LLMs with Refined Rotationcs.LG
DFRot improves LLMs by reducing outlier and massive activation effects.
problem Reducing outlier and massive activation effects in rotated LLMs.
method Weighted loss function and orthogonal Procrustes transforms for rotation matrix refinement.
result DFRot achieves dual free (Outlier-Free and Massive Activation-Free) with significant improvements in perplexity.
An Asynchronous Distributed Expectation Maximization Algorithm For Massive Data: The DEM Algorithmstat.CO
The family of Expectation-Maximization (EM) algorithms provides a general approach to fitting flexible models for large and complex data. The expectation (E) step of EM-type algorithms is time-consuming in massive data applications because it requires multiple passes through the full data. We address this problem by pr…
TrustNet robustly learns noise patterns from trusted data to improve weakly-supervised classification.
problem Robustness to label noise in weakly-supervised learning.
method TrustNet learns noise patterns from trusted data, then trains a robust classifier using these patterns.
result TrustNet outperforms state-of-the-art methods in robustness to various noise patterns.
Massively parallel RWS improves inference in complex models.
problem Exponential sample requirement for effective importance weighting.
method Draws samples of all latent variables and individually reasons through all combinations.
result Significant improvements over standard RWS.
Here we prove the existence of a new type of the world-sheet string singularities - the cusps that are stable during the finite time. These singularities make the emission of the captured massive quantum particle possible in the frames of the author's model suggested earlier. In aggregate, we have a new mechanism of qu…
We prove a Goldberg-Sachs theorem in dimension three. To be precise, given a three-dimensional Lorentzian manifold satisfying the topological massive gravity equations, we provide necessary and sufficient conditions on the tracefree Ricci tensor for the existence of a null line distribution whose orthogonal complement …