Advanced video classification systems decode video frames to derive the necessary texture and motion representations for ingestion and analysis by spatio-temporal deep convolutional neural networks (CNNs). However, when considering visual Internet-of-Things applications, surveillance systems and semantic crawlers of la…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
169,051 papers · 148 categories
Trend · papers per month
48 results for “data ingestion”
This paper uses LLMs to improve equity stock ratings by ingesting diverse financial and news data.
problem Challenges in traditional stock rating methods, including data overload, inconsistencies, and delayed reactions.
method Application of LLMs to generate multi-horizon stock ratings using various datasets.
result LLMs enhance the accuracy and consistency of stock ratings, outperforming traditional methods in forward returns.
AI pipeline simplifies AI deployment on embedded devices.
problem Training and deployment of custom AI solutions on embedded devices requires expertise and integration barriers.
method Modular AI pipeline integrating data, algorithms, and deployment tools.
result LPDNN consistently outperforms other deployment frameworks on embedded platforms.
Dynamic Mortality Risk Predictions in Pediatric Critical Care Using Recurrent Neural Networksstat.ML
Viewing the trajectory of a patient as a dynamical system, a recurrent neural network was developed to learn the course of patient encounters in the Pediatric Intensive Care Unit (PICU) of a major tertiary care center. Data extracted from Electronic Medical Records (EMR) of about 12000 patients who were admitted to the…
Labeling training data is one of the most costly bottlenecks in developing machine learning-based applications. We present a first-of-its-kind study showing how existing knowledge resources from across an organization can be used as weak supervision in order to bring development time and cost down by an order of magnit…
VeLO learns versatile optimizers from deep learning tasks.
problem Training deep learning models with hand-designed optimizers.
method Meta-training a neural network optimizer on a wide variety of optimization tasks.
result The learned optimizer automatically adapts to different optimization tasks without hyperparameter tuning.
EcoCast predicts biodiversity risks using satellite data and citizen science records.
problem Unprecedented shifts in species distributions due to climate change and habitat loss.
method Spatio-temporal model using sequence-based transformers and continual learning.
result Promising improvements in forecasting bird species distributions compared to Random Forest.
A Unified AI System For Data Quality Control and DataOps Management in Regulated Environmentsq-fin.CP
Unified AI system for data quality control and governance in regulated environments.
problem Isolated data quality control steps in existing systems.
method AI-driven framework integrating rule-based, statistical, and AI methods.
result Empirical gains in anomaly detection, reduced manual remediation, improved auditability.
ACI converts call center conversations into actionable data.
problem Real-time spoken language understanding for call center conversations.
method Combines speech recognition, entity and intent recognition, and a business rules engine.
result ACI converts live audio into structured events for real-time supervision and assistance.
Microsoft develops a time-series anomaly detection service.
problem Real-time monitoring of various metrics in applications and services.
method Pipeline consisting of data ingestion, experimentation platform, and online compute; novel algorithm combining Spectral Residual and Convolutional Neural Network.
result Superior experimental results compared to state-of-the-art baselines on public and Microsoft production data.
Paper proposes a new Autoencoder for robustly encoding white matter streamlines.
problem Limited Autoencoder architectures ignore global streamline geometry and lack interpretability.
method Introduces Differentiable Vector Quantized Variational Autoencoder (D-VQ-VAE) for entire streamline bundles.
result Demonstrates superior performance in encoding and synthesis compared to state-of-the-art Autoencoders.
Dance Dance Revolution (DDR) is a popular rhythm-based video game. Players perform steps on a dance platform in synchronization with music as directed by on-screen step charts. While many step charts are available in standardized packs, players may grow tired of existing charts, or wish to dance to a song for which no …
MPLP learns neural network weights by treating operations as message-passing agents.
problem Training neural networks using gradient-based methods.
method MPLP abstracts neural network operations as message-passing agents, updating internal states and passing messages.
result MPLP outperforms traditional gradient-based methods on simple feed-forward neural networks.
Kolmogorov-Arnold network improves GW catalog posterior construction.
problem Efficiently constructing posterior distributions for GW catalogs.
method Using the Kolmogorov-Arnold network to create lightweight neural density estimators.
result Kolmogorov-Arnold network achieves superior interpretability and accuracy in posterior construction.
Transformer with denoising diffusion improves probabilistic density estimation.
problem Estimating non-Gaussian and multimodal probability distributions for regression problems.
method Training a denoising diffusion head on top of a Transformer model.
result The model provides reasonable probability density estimation for high-dimensional inputs.
Karate Club simplifies graph mining for unsupervised learning.
problem Efficient unsupervised learning on graphs.
method API-oriented Python framework combining 30+ algorithms.
result Karate Club outperforms in learning performance and speed.
Improved aircraft structure prediction using derivative-enhanced sparse Cholesky GP method.
problem Accurate real-time prediction of aircraft structure performance.
method Combining derivative data with a modified dynamic sparse Cholesky linear system solver.
result Improved prediction accuracy of aircraft structure performance.
The paper presents anomaly detection in time series data using InfluxDB and Python.
problem Anomalous data points in time series data affect decision making in water and environmental systems.
method Data cleaning, cost-sensitive machine learning (Logistic Regression, Random Forest, SVM), feature selection, and InfluxDB integration.
result Random Forest outperformed other models in detecting anomalies.
Automates fairness and accuracy optimization in deep learning models for tabular data.
problem Improving fairness and accuracy in neural models for tabular data.
method Employed multi-objective Neural Architecture Search (NAS) and Hyperparameter Optimization (HPO) to find new models.
result Jointly optimized architectures that consistently outperform single-objective fairness mitigation methods.
Currently, approximately 30% of epileptic patients treated with antiepileptic drugs (AEDs) remain resistant to treatment (known as refractory patients). This project seeks to understand the underlying similarities in refractory patients vs. other epileptic patients, identify features contributing to drug resistance acr…
Smart app tracks relapse history and predicts relapse based on spatial-temporal factors.
problem Relapse prevention for alcohol and tobacco addiction users.
method Records user profiles, tracks relapse history, uses machine learning for prediction, and recommends activities.
result Predictive machine learning algorithms help in preventing relapse.
Reliable identification of molecular biomarkers is essential for accurate patient stratification. While state-of-the-art machine learning approaches for sample classification continue to push boundaries in terms of performance, most of these methods are not able to integrate different data types and lack generalization…
Auxiliary Tuning adapts pre-trained models for novel tasks efficiently.
problem Adapting pre-trained models for new tasks efficiently.
method Supplementing pre-trained model with an auxiliary model that shifts output distribution.
result Achieved similar results to training from scratch with fewer resources.
Isthmus platform simplifies ML/AI integration in healthcare.
problem Challenges in deploying ML models in healthcare due to data quality, regulatory, and security issues.
method Turnkey, cloud-based platform addressing data quality, clinical relevance, and regulatory compliance.
result Reduces time to market for operationalizing ML/AI in healthcare.
Deep learning predicts fit for fashion e-commerce.
problem Predicting correct fit for customer satisfaction and cost reduction.
method Deep learning content-collaborative approach using customer and article embeddings.
result Significant improvement over state-of-the-art methods.
A new method corrects bias in machine learning for trading by filtering out non-executable prices.
problem Contaminated data from price limits causes misleading performance metrics in trading models.
method Mask-first design with tradability filters, GPU optimization, and penalized losses.
result The system achieves Sharpe ratios of 1.63 on real data, significantly outperforming existing methods.
Bayesian encoding improves lead scoring for WeWork using conjugate models.
problem Encoding high-cardinality categorical features for machine learning.
method Conjugate Bayesian models for categorical features, ensemble learning.
result AUC improved from 0.87 to 0.97 for WeWork's lead scoring engine.
Principal component analysis (PCA) is one of the most powerful tools in machine learning. The simplest method for PCA, the power iteration, requires full-data passes to recover the principal component of a matrix with eigen-gap . Lanczos, a significantly more complex method, achieves an accelerated…
This paper tackles delayed feedback in continuous training for CTR prediction, improving model performance by 3%.
problem Delayed feedback in CTR prediction leads to inferior performance and user experience.
method Comparing 5 loss functions and models in offline and online settings.
result Proposed methods outperform previous state-of-the-art by 3% relative cross entropy (RCE).
Deep learning identifies transcriptomic patterns and cell types associated with SARS-CoV-2 infection and COVID-19 severity.
problem Understanding how SARS-CoV-2 varies in infecting and causing severe COVID-19.
method Developed a new approach to generating self-supervised edge features, using Graph Attention Networks (GAT) and Set Transformer.
result Achieved state-of-the-art performance in predicting disease state of individual cells using single-cell RNA sequencing data.
CheXpert++: Approximating the CheXpert labeler for Speed,Differentiability, and Probabilistic Outputcs.LG
CheXpert++ improves CheXpert's accuracy and usability for medical radiology reports.
problem Infeasibility of obtaining ground truth labels for medical data.
method BERT-based approximation of CheXpert, addressing speed, differentiability, and probabilistic output.
result Achieves 99.81% parity with CheXpert, significantly faster, differentiable, and probabilistic.
New deep learning model estimates scattering timescale of FRBs efficiently.
problem Estimating scattering timescale of fast radio bursts (FRBs) is a bottleneck.
method Multimodal Transformer Based Generic Mixture Density Network (MT-GMDN) that ingests dynamic spectrum and timeseries profile.
result Achieves 94% on expected value of for measurable scattering.
Develops a new framework for perpetual futures on binary prediction markets.
problem Lack of effective risk management in perpetual futures on binary prediction markets.
method PIRAP framework with six components: index estimator, margin sizing, leverage, funding rule, halt protocol, and eligibility framework.
result Mixed results from empirical evaluation, with some pre-registered floors passing and others failing.
Study of Polymarket's prediction market microstructure using tick-level order book data.
problem Understanding the microstructure of decentralized prediction markets.
method Analysis of a continuous tick-level order book feed and on-chain trade records.
result Trade direction inferred from Polymarket's public order-book feed disagrees with on-chain data in ~59% of cases.
With the proliferation of social media, fashion inspired from celebrities, reputed designers as well as fashion influencers has shortened the cycle of fashion design and manufacturing. However, with the explosion of fashion related content and large number of user generated fashion photos, it is an arduous task for fas…
This technique learns interpretable models by encoding the training distribution as a Dirichlet Process and using uncertainty scores as an oracle.
problem Creating small, interpretable models that are still accurate.
method Exploits a Dirichlet Process to encode the training distribution, uses Bayesian Optimization for parameters, and projects data to one dimension using an uncertainty oracle.
result Improves accuracy and size trade-off, applicable across different model families.
Zero-Copy Semantic Contagion: An In-Memory Streaming Architecture for Evolving Attention Graphscs.LG
Zero-Copy Architecture Detects Cross-Company Financial Signals Instantly.
problem Financial models miss cross-company disruptions due to static data.
method Heterogeneous Rust-Python streaming architecture that maps cross-company attention as a continuous-time graph.
result Zero-copy parsing and inference process delivers real-time cross-company signal detection.
T-EMDE bridges the heterogeneity gap between image and text modalities.
problem Finding similarities between image and text modalities with non-related feature spaces.
method Inspired by EMDE, T-EMDE uses sketches for multimodal operations, avoiding self-attention's quadratic complexity.
result T-EMDE achieves state-of-the-art results and reduces model latency.
Data preprocessing improves data quality for robust data mining.
problem Noisy and incomplete data hinders data mining models.
method Overview of data cleaning, transformation, and preprocessing methods.
result Preprocessing significantly affects data mining model performance.
Big data sets must be carefully partitioned into statistically similar data subsets that can be used as representative samples for big data analysis tasks. In this paper, we propose the random sample partition (RSP) data model to represent a big data set as a set of non-overlapping data subsets, called RSP data blocks,…
A new method for handling imbalanced big data using ensembles and smart data.
problem Imbalanced data distribution in big data scenarios.
method Smart Data driven Decision Trees Ensemble (SD_DeTE) methodology.
result SD_DeTE outperforms Random Forest in handling imbalanced binary classification problems in big data.
Prevents sensitive data generation in diffusion models using labeled and unlabeled data.
problem Generating sensitive data in diffusion models using unlabeled data.
method Positive-Unlabeled Diffusion Models, approximating ELBO with labeled and unlabeled data.
result Prevents the generation of sensitive data without compromising image quality.
Study reveals Data Shapley's inconsistent performance in data selection tasks.
problem Inconsistency of Data Shapley's performance in data selection across different settings.
method Hypothesis testing framework and identification of utility functions.
result Data Shapley's performance is no better than random selection without specific constraints.
PRRO generates synthetic tabular data that improves SL performance and class distribution.
problem Low SL utility of synthetic data due to class imbalance and overlooked data relationships.
method Data pruning and column reordering to optimize SL utility.
result Synthetic data generated with PRRO enhances predictive performance and class distribution.
Defines data science as a natural ecosystem with challenges and missions.
problem Challenges and missions in data science due to 5D complexities and data life cycle phases.
method Systemic and data-centric view of data science as a fusion of data universe and its challenges, formalizing a general-purpose architecture.
result Essential data science as a natural ecosystem integrating specific disciplines and high-impact applications.
Synthetic data enhances analytics but requires careful volume management.
problem Accuracy of statistical methods on synthetic data vs. raw data.
method Synthetic Data Generation for Analytics framework using tabular diffusion models.
result Error rate decreases with more synthetic data but may stabilize or increase.
This paper introduces C-DSL to improve data mining outcomes by considering context.
problem Data collection ambiguities, data imbalance, hidden biases, lack of domain info, and data incompleteness.
method Developed Context-Driven Data Science Lifecycle (C-DSL) to address data quality issues.
result Tangible improvements to data mining outcomes were achieved through C-DSL.
Proposes using probabilistic models for privacy-preserving synthetic data.
problem Designing high-quality synthetic data for privacy preservation.
method Formulate the problem through probabilistic modelling, choosing a model for the data.
result Statistical discoveries can be reliably reproduced from synthetic data.