DPASF stream preprocesses Big Data streams efficiently.
problem Efficient preprocessing of streaming Big Data.
method Implemented six preprocessing algorithms in Apache Flink.
result Preprocessing improves data accuracy in streaming Big Data.
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
169,051 papers · 148 categories
Trend · papers per month
48 results for “Apache Flink”
The AMIDST Toolbox is a software for scalable probabilistic machine learning with a spe- cial focus on (massive) streaming data. The toolbox supports a flexible modeling language based on probabilistic graphical models with latent variables and temporal dependencies. The specified models can be learnt from large data s…
Paper optimizes a big data and ML risk monitoring system for financial markets.
problem Traditional risk monitoring methods are inadequate for modern financial markets due to data complexity and volume.
method Four-layer architecture integrating big data and advanced ML algorithms (LSTM, RF, GB).
result Significantly enhances efficiency and accuracy in risk management, especially in market crash risk detection.
Spark implementation for distributed function minimization.
problem Optimization of functions in distributed computing environments.
method Gradient and quasi-Newton methods on Apache Spark.
result Scalable solution for classification and regression problems.
Unified machine learning framework for deep learning and web services.
problem Deep learning and web service challenges.
method MMLSpark expands Spark to deep learning, micro-service orchestration, etc.
result Unified API for deep learning and web services.
With the spreading prevalence of Big Data, many advances have recently been made in this field. Frameworks such as Apache Hadoop and Apache Spark have gained a lot of traction over the past decades and have become massively popular, especially in industries. It is becoming increasingly evident that effective big data a…
GluonCV and GluonNLP simplify deep learning for CV and NLP.
problem Lack of easy-to-use deep learning tools for CV and NLP.
method Developed modular APIs and pre-trained models for rapid prototyping.
result Facilitates rapid prototyping and reproducible research in CV and NLP.
Apache Spark is a popular open-source platform for large-scale data processing that is well-suited for iterative machine learning tasks. In this paper we present MLlib, Spark's open-source distributed machine learning library. MLlib provides efficient functionality for a wide range of learning settings and includes sev…
PyODDS automates outlier detection for new data sources.
problem Manual outlier detection is inefficient and domain-specific.
method Automated end-to-end system with Apache Spark and database support.
result PyODDS optimizes outlier detection pipelines automatically.
A new algorithm predicts periodic time series data efficiently in cloud environments.
problem Efficiently identifying and predicting periodic patterns in large-scale time-series data.
method Proposes a Periodicity-based Parallel Time Series Prediction (PPTSP) algorithm using TSDCA, MTSPPR, and PTSP methods.
result Significant improvements in prediction accuracy and performance compared to existing algorithms.
BreachRadar detects points-of-compromise in bank transactions to prevent fraud.
problem Detecting and preventing bank transaction fraud caused by data breaches.
method A distributed alternating algorithm that assigns probabilities to different locations being compromised.
result BreachRadar achieves over 90% precision and recall in detecting compromised cards.
DiCFS improves CFS for big data, handling large datasets efficiently.
problem Efficient feature selection for large datasets in big data.
method Distributed CFS (DiCFS) using Apache Spark for scalability and parallel processing.
result DiCFS outperforms WEKA in terms of time-efficiency and scalability.
JAMPI improves matrix multiplication in Spark, boosting performance by up to 24%.
problem Efficiently performing matrix multiplication in Spark.
method Combining asynchronous network IO, auto-vectorization, and barrier execution mode.
result Up to 24% performance increase in distributed matrix multiplication.
Training deep networks is expensive and time-consuming with the training period increasing with data size and growth in model parameters. In this paper, we provide a framework for distributed training of deep networks over a cluster of CPUs in Apache Spark. The framework implements both Data Parallelism and Model Paral…
GraSPy simplifies graph analysis in Python.
problem Analyzing and understanding graphs.
method Scikit-learn compliant API for statistical inference and machine learning.
result Flexible algorithms for graph statistics.
DADApy analyzes high-dimensional data manifolds in Python.
problem Analyzing complex, high-dimensional data.
method Estimating intrinsic dimension, density, clustering, comparing distance metrics.
result Effective analysis of data manifolds in Python.
TailedTS: Benchmark Dataset for Heavy-Tailed Time Series Prediction and Periodicity Quantificationcs.LG
TailedTS dataset benchmarks heavy-tailed time series forecasting and periodicity quantification.
problem Benchmarking robustness of time series models under heavy-tailed distributions.
method Derived from Wikipedia page views, introduces periodicity quantification and robust loss functions.
result Standard Gaussian models degrade on high-volume page categories, while robust alternatives perform consistently.
In this paper we present a new algorithm for computing a low rank approximation of the product by taking only a single pass of the two matrices and . The straightforward way to do this is to (a) first sketch and individually, and then (b) find the top components using PCA on the sketch. Our algori…
torchsom simplifies SOMs in PyTorch with GPU acceleration and scikit-learn API.
problem Efficient implementation and usability of SOMs in PyTorch.
method PyTorch backend, GPU acceleration, scikit-learn API, 90% test coverage.
result Ease of use and scalability for SOMs in PyTorch.
A {\em blink} is a plane graph with an arbitrary bipartition of its edges. As a consequence of a recent result of Martelli, I show that the homeomorphisms classes of closed oriented 3-manifolds are in 1-1 correspondence with specific classes of blinks. In these classes, two blinks are equivalent if they are linked by a…
Python toolbox for causal structure learning from data.
problem Causal structure learning from data.
method Generates data from simulators or real-world datasets, learns causal structure, evaluates graphs, and includes gradient-based methods.
result Convenience and efficiency in causal discovery with GPU acceleration.
TODS automates time series outlier detection with customizable pipelines.
problem Automated detection of outliers in time series data.
method Modular system with 70 primitives for data processing, time series analysis, and detection algorithms. GUI and data-driven searcher for pipeline design.
result Automated discovery and construction of effective outlier detection pipelines.
We consider the problem of learning a high-dimensional but low-rank matrix from a large-scale dataset distributed over several machines, where low-rankness is enforced by a convex trace norm constraint. We propose DFW-Trace, a distributed Frank-Wolfe algorithm which leverages the low-rank structure of its updates to ac…
In this paper, a neural network-based stock price prediction and trading system using technical analysis indicators is presented. The model developed first converts the financial time series data into a series of buy-sell-hold trigger signals using the most commonly preferred technical analysis indicators. Then, a Mult…
Signatory calculates signature and logsignature transforms efficiently on CPU and GPU.
problem Efficient computation of signature and logsignature transforms for machine learning.
method CPU and GPU parallelism, backpropagation, efficient precomputation strategies, algorithmic improvements.
result Substantial speedups on CPU and GPU, including real-world applications.
The People's Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usagecs.LG
Large speech dataset for commercial use with 9.98% word error rate.
problem Creating a diverse speech recognition dataset for commercial purposes.
method Internet search for licensed audio data with transcriptions, training model on the dataset.
result Model trained on dataset achieves 9.98% word error rate on Librispeech's test-clean test set.
Critically ill patients in regular wards are vulnerable to unanticipated clinical dete- rioration which requires timely transfer to the intensive care unit (ICU). To allow for risk scoring and patient monitoring in such a setting, we develop a novel Semi- Markov Switching Linear Gaussian Model (SSLGM) for the inpatient…
TorchBeast simplifies RL research in PyTorch.
problem Conducting scalable RL research with minimal programming knowledge.
method Pure-Python and C++ implementation for parallel training; OpenAI Gym interface.
result Performs on-par with IMPALA on Atari.
Advbox generates adversarial examples to test neural network robustness.
problem Vulnerability of neural networks to adversarial examples.
method Tool to create adversarial examples in multiple frameworks.
result Benchmarking robustness of machine learning models.
secml is a Python library for secure and explainable machine learning.
problem Evaluate and understand security and explainability of machine learning models.
method Implement attacks and explainability methods for various algorithms.
result Demonstrates how quickly models fail under adversarial perturbations and visualizes influential features.
PyHHMM is a Python library for HHMMs with advanced features.
problem Handling heterogeneous observation models and missing data in HMMs.
method Object-oriented Python implementation with advanced features.
result PyHHMM supports a heterogeneous observation model and missing data inference.
DALC customizes LSTM models for detectors in large-scale traffic networks.
problem Fine-grained traffic prediction for large-scale transportation networks.
method Formulated as a finite Markov decision process, introduced ALC algorithm for automatic customization, and developed DALC for distributed customization.
result DALC provides higher prediction accuracy than Apache Spark MLlib approaches.
We describe Sockeye (version 1.12), an open-source sequence-to-sequence toolkit for Neural Machine Translation (NMT). Sockeye is a production-ready framework for training and applying models as well as an experimental platform for researchers. Written in Python and built on MXNet, the toolkit offers scalable training a…
Training modern deep learning models requires large amounts of computation, often provided by GPUs. Scaling computation from one GPU to many can enable much faster training and research progress but entails two complications. First, the training library must support inter-GPU communication. Depending on the particular …
With large volumes of health care data comes the research area of computational phenotyping, making use of techniques such as machine learning to describe illnesses and other clinical concepts from the data itself. The "traditional" approach of using supervised learning relies on a domain expert, and has two main limit…
Supervised learning algorithms are nowadays successfully scaling up to datasets that are very large in volume, leveraging the potential of in-memory cluster-computing Big Data frameworks. Still, massive datasets with a number of large-domain categorical features are a difficult challenge for any classifier. Most off-th…
ICU mortality scoring systems attempt to predict patient mortality using predictive models with various clinical predictors. Examples of such systems are APACHE, SAPS and MPM. However, most such scoring systems do not actively look for and include interaction terms, despite physicians intuitively taking such interactio…
WRSE predicts dynamic survival distributions in ICU patients.
problem Dynamic assessment of ICU patient mortality risk.
method Non-parametric weighted-resolution ensemble model combining binary classifiers.
result Competitive results with state-of-the-art models, reducing training time.
We describe DyNet, a toolkit for implementing neural network models based on dynamic declaration of network structure. In the static declaration strategy that is used in toolkits like Theano, CNTK, and TensorFlow, the user first defines a computation graph (a symbolic representation of the computation), and then exampl…
Specialized Deep Learning (DL) acceleration stacks, designed for a specific set of frameworks, model architectures, operators, and data types, offer the allure of high performance while sacrificing flexibility. Changes in algorithms, models, operators, or numerical systems threaten the viability of specialized hardware…
PROFET builds DBNs from ODEs, handling uncertainty in data and models.
problem Handling uncertainty in both data and models for DBN construction.
method Automatic DBN construction from ODEs, adaptive-time particle filtering.
result PROFET automates DBN construction and inference from ODE models.
Tribuo integrates ML models with provenance in Java.
problem Tracking and ensuring ML models behave appropriately as they grow in number and complexity.
method Integrates model training, inference, type-safety, runtime checking, and automatic provenance recording into a single Java library.
result Automatic recording of full processing pipeline and training details inside the model object.
Big data trend has enforced the data-centric systems to have continuous fast data streams. In recent years, real-time analytics on stream data has formed into a new research field, which aims to answer queries about what-is-happening-now with a negligible delay. The real challenge with real-time stream data processing …
A Disease Diagnosis and Treatment Recommendation System Based on Big Data Mining and Cloud Computingcs.LG
System recommends disease treatments based on big data and cloud computing.
problem Inaccurate disease classification and treatment recommendations due to complex symptoms and multi-pathogenesis.
method DPCA for disease-symptom clustering, Apriori for D-D and D-T rules, parallel Apache Spark implementation.
result Effective disease-symptom clustering and accurate treatment recommendations for inexperienced doctors.
Parallel Protein Community Detection in Large-scale PPI Networks Based on Multi-source Learningcs.SI
Paper proposes MLPCD for protein community detection in large PPI networks.
problem Identifying reliable protein communities from large-scale PPI networks.
method Integrates Gene Expression Data and uses Multi-source Learning with cloud computing.
result Demonstrates superior performance compared to existing methods.
Over the past decades, both critical care and cancer care have improved substantially. Due to increased cancer-specific survival, we hypothesized that both the number of cancer patients admitted to the ICU and overall survival have increased since the millennium change. MIMIC-III, a freely accessible critical care data…
In recent years, analyzing task-based fMRI (tfMRI) data has become an essential tool for understanding brain function and networks. However, due to the sheer size of tfMRI data, its intrinsic complex structure, and lack of ground truth of underlying neural activities, modeling tfMRI data is hard and challenging. Previo…
Study builds models to predict post-cardiac arrest outcomes using patient data.
problem Lack of accurate prognostication methods for patients resuscitated from cardiac arrest.
method Integrated electronic health records (EHR) and physiological time series (PTS) data to train machine learning classifiers.
result Combined EHR-PTS24 models outperformed models using either EHR or PTS24 alone in predicting survival and neurological outcomes.