Record linkage involves merging records in large, noisy databases to remove duplicate entities. It has become an important area because of its widespread occurrence in bibliometrics, public health, official statistics production, political science, and beyond. Traditional linkage methods directly linking records to one…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “Bayesian record linkage”
Bayesian approach for multifile record linkage and duplicate detection.
problem Challenges in merging overlapping datafiles with duplicates.
method Bayesian approach with novel partition representation and loss functions.
result Proposes a flexible prior for partitions and uncertain unresolved portions.
The bipartite record linkage task consists of merging two disparate datafiles containing information on two overlapping sets of entities. This is non-trivial in the absence of unique identifiers and it is important for a wide variety of applications given that it needs to be solved whenever we have to combine informati…
Motivated by the need of the linking records across various databases, we propose a novel graphical model based classifier that uses a mixture of Poisson distributions with latent variables. The idea is to derive insight into each pair of hypothesis records that match by inferring its underlying latent rate of error us…
Entity resolution (ER; also known as record linkage or de-duplication) is the process of merging noisy databases, often in the absence of unique identifiers. A major advancement in ER methodology has been the application of Bayesian generative models, which provide a natural framework for inferring latent entities with…
The task of matching co-referent records is known among other names as rocord linkage. For large record-linkage problems, often there is little or no labeled data available, but unlabeled data shows a reasonable clear structure. For such problems, unsupervised or semi-supervised methods are preferable to supervised met…
In record linkage (RL), or exact file matching, the goal is to identify the links between entities with information on two or more files. RL is an important activity in areas including counting the population, enhancing survey frames and data, and conducting epidemiological and follow-up studies. RL is challenging when…
In this short paper, the Electre Tri-Machine Learning Method, generally used to solve ordinal classification problems, is proposed for solving the Record Linkage problem. Preliminary experimental results show that, using the Electre Tri method, high accuracy can be achieved and more than 99% of the matches and nonmatch…
Two solutions for multi-modal record linkage using Deep Learning inspired by Visual Question Answering.
problem Matching records from multiple sources representing the same entity.
method Two fusion modules: Recurrent Neural Network + Convolutional Neural Network and Stacked Attention Network. A Siamese Neural Network computes similarity.
result Recurrent Neural Network + Convolutional Neural Network fusion module outperforms a simple model.
We present a probabilistic method for linking multiple datafiles. This task is not trivial in the absence of unique identifiers for the individuals recorded. This is a common scenario when linking census data to coverage measurement surveys for census coverage evaluation, and in general when multiple record-systems nee…
This article reviews entity resolution methods and their applications.
problem Integrating information from multiple sources to clean and accurately link records.
method Clustering, semi- and fully supervised methods, canonicalization.
result Modern probabilistic record linkage has been foundational.
This paper develops a non-Bayesian methodology to analyze the time-varying structure of international linkages and market efficiency in G7 countries. We consider a non-Bayesian time-varying vector autoregressive (TV-VAR) model, and apply it to estimate the joint degree of market efficiency in the sense of Fama (1970, 1…
New framework estimates staged tree models using hierarchical clustering on the probability simplex.
problem Estimating staged tree models with context-specific dependencies.
method Hierarchical clustering on the probability simplex, using simplex-based divergences and linkage methods.
result Total Variation divergence with Ward.D2 linkage produces staged trees with better model fit, structure recovery, and computational efficiency.
Entity resolution (ER) is the task of identifying records belonging to the same entity (e.g. individual, group) across one or multiple databases. Ironically, it has multiple names: deduplication and record linkage, among others. In this paper we survey metrics used to evaluate ER results in order to iteratively improve…
This paper explores combinatorial optimization for problems of max-weight graph matching on multi-partite graphs, which arise in integrating multiple data sources. Entity resolution-the data integration problem of performing noisy joins on structured data-typically proceeds by first hashing each record into zero or mor…
We present a database of parliamentary debates that contains the complete record of parliamentary speeches from Dáil Éireann, the lower house and principal chamber of the Irish parliament, from 1919 to 2013. In addition, the database contains background information on all TDs (Teachta Dála, members of parliament), such…
We develop a structural default model for interconnected financial institutions in a probabilistic framework. For all possible network structures we characterize the joint default distribution of the system using Bayesian network methodologies. Particular emphasis is given to the treatment and consequences of cyclic fi…
Versatile linkage: a family of space-conserving strategies for agglomerative hierarchical clusteringstat.ME
Agglomerative hierarchical clustering can be implemented with several strategies that differ in the way elements of a collection are grouped together to build a hierarchy of clusters. Here we introduce versatile linkage, a new infinite system of agglomerative hierarchical clustering strategies based on generalized mean…
New method for estimating firm linkages using CVLs and QCML.
problem Estimating firm linkages for profitable trading strategies.
method Characteristic Vector Linkages (CVLs) and Quantum Cognition Machine Learning (QCML).
result QCML similarity outperforms Euclidean similarity in constructing profitable trading strategies.
The geometric, topological, and symplectic properties of moduli spaces (spaces of configurations modulo rotations and translations) of polygonal linkages have been studied by Kapovich, Millson, and Kamiyama, et. al. One can form a polygonal linkage by taking two free linkages and identifying initial and terminal vertic…
This paper studies the configuration space of all possible positions of a linkage in R^n. For example, it shows that for every compact algebraic set, there is a linkage whose configuration space is analytically isomorphic to a finite number of copies of the algebraic set. If flexible edges are allowed, any compact set …
Regularization for Shuffled Data Problems via Exponential Family Priors on the Permutation Groupstat.ML
New method uses exponential family priors to handle shuffled data problems.
problem Handling mismatch errors in record linkage of two data files.
method Flexible exponential family prior on the permutation group for regularization.
result The proposed method outperforms competing methods in synthetic and real data.
Benchmarking Minimax Linkagestat.ML
Minimax linkage was first introduced by Ao et al. [3] in 2004, as an alternative to standard linkage methods used in hierarchical clustering. Minimax linkage relies on distances to a prototype for each cluster; this prototype can be thought of as a representative object in the cluster, hence improving the interpretabil…
Cryptocurrencies are becoming more linked in their returns and volatilities.
problem Understanding the increasing interconnectivity of cryptocurrencies.
method Examined market linkages using returns and volatilities, applied various methodologies.
result Significant increase in market linkages for both returns and volatilities.
Data of practical interest - such as personal records, transaction logs, and medical histories - are sequential collections of events relevant to a particular source entity. Recent studies have attempted to link sequences that represent a common entity across data sets to allow more comprehensive statistical analyses a…
Paper addresses limitations of traditional hierarchical clustering methods.
problem Traditional hierarchical clustering methods face limitations in binary trees and ultrametrics.
method Introduces the notion of a valid hierarchy and a two-step algorithm to construct a binary tree and prune it to enforce validity.
result Proposes a method to recover the finest valid hierarchy, which is not constrained to binary structures.
Moduli spaces of planar polygonal linkages admit a cell structure which can be realized as a surgery on the permutohedron. We present a 3D visualization of the result of the surgery for all types of non-degenerate pentagonal linkages.
We prove realizability theorems for vector-valued polynomial mappings, real-algebraic sets and compact smooth manifolds by moduli spaces of planar linkages. We also establish a relation between universality theorems for moduli spaces of mechanical linkages and projective arrangements.
Paper analyzes a three-loop linkage, showing it's overconstrained and shaky.
problem Analyzing a three-loop spatial linkage's degree of freedom and configuration space.
method Local analysis of differential degrees of freedom, computation of kinematic tangent cone, and c-space approximation.
result The linkage has a finite degree of freedom 3 and is locally a smooth manifold, making it shaky.
A mechanical linkage is a mechanism made of rigid rods linked together by flexible joints, in which some vertices are fixed and others may move. The partial configuration space of a linkage is the set of all the possible positions of a subset of the vertices. We characterize the possible partial configuration spaces of…
A closed linkage mechanism in three-dimensional space is an object comprising rigid bodies connected with hinges in a circular form like a rosary. Such linkages include Bricard6R and Bennett4R. To design such a closed linkage, it is necessary to solve a high-degree algebraic equation, which is generally difficult. In t…
A linkage is a finite graph with lengths assigned to each edge. A planar realization is a map to the plane which preserves edge lengths. It can be thought of as a mechanical device formed from stiff rods and rotating joints. We look at the configuration space of all planar realizations of a linkage (following work of K…
In supervised clustering, standard techniques for learning a pairwise dissimilarity function often suffer from a discrepancy between the training and clustering objectives, leading to poor cluster quality. Rectifying this discrepancy necessitates matching the procedure for training the dissimilarity function to the clu…
Crimes emerge out of complex interactions of human behaviors and situations. Linkages between crime incidents are highly complex. Detecting crime linkage given a set of incidents is a highly challenging task since we only have limited information, including text descriptions, incident times, and locations. In practice,…
A mechanical linkage is a mechanism made of rigid rods linked together by flexible joints, in which some vertices are fixed and others may move. The partial configuration space of a linkage is the set of all the possible positions of a subset of the vertices. We characterize the possible partial configuration spaces of…
Study explores financial market linkages between Japan and US markets.
problem Inconsistency in empirical studies regarding financial market causal linkages.
method Causal discovery methods including VAR-LiNGAM and LPCMCI with domain knowledge.
result VAR-LiNGAM reveals causal influences among financial markets, while LPCMCI identifies potential latent confounders.
Fetal ECG (FECG) telemonitoring is an important branch in telemedicine. The design of a telemonitoring system via a wireless body-area network with low energy consumption for ambulatory use is highly desirable. As an emerging technique, compressed sensing (CS) shows great promise in compressing/reconstructing data with…
User identity linkage is a task of recognizing the identities of the same user across different social networks (SN). Previous works tackle this problem via estimating the pairwise similarity between identities from different SN, predicting the label of identity pairs or selecting the most relevant identity pair based …
New bounds on the cohesion of complete-link and other linkage methods for agglomeration clusteringcs.LG
New bounds improve linkage methods for clustering, distinguishing complete-link from single-link.
problem Improving bounds on linkage methods for clustering quality.
method Developed new bounds for complete-link and average-link methods in agglomeration clustering.
result Separated complete-link from single-link in terms of approximation for diameter.
We study configuration spaces of linkages whose underlying graph are polygons with diagonal constrains, or more general, partial two-trees. We show that (with an appropriate definition) the oriented area is a Bott-Morse function on the configuration space. Its critical points are described and Bott-Morse indices are co…
Bayesian entity resolution merges together multiple, noisy databases and returns the minimal collection of unique individuals represented, together with their true, latent record values. Bayesian methods allow flexible generative models that share power across databases as well as principled quantification of uncertain…
Hierarchical clustering uses OWA operators to generalize linkage methods and avoid dendrogram inversions.
problem Avoiding unaesthetic inversions in hierarchical clustering dendrograms.
method OWA-based linkages combined with the Lance-Williams formula and conditions on weight generators.
result Conditions for weight generators to produce dendrograms without inversions.
Series Parallel Linkagesmath.GT
We study spaces of realisations of linkages (weighted graphs) whose underlying graph is a series parallel graph. In particular, we describe an algorithm for determining whether or not such spaces are connected.
Bayesian method suppresses low-frequency pulses in audio recordings.
problem Suppressing long pulses caused by mechanical defects in audio recordings.
method Bayesian approach using Gaussian Process for pulse location, signal interpolation, and tail estimation.
result Perceptual results similar to previous methods, performs well on naturally degraded signals.
A linkage mechanism consists of rigid bodies assembled by joints which can be used to translate and transfer motion from one form in one place to another. In this paper, we are particularly interested in a family of spacial linkage mechanisms which consist of -copies of a rigid body joined together by hinges to form…
Federated learning for Bayesian clustering of large datasets.
problem Bayesian model-based clustering of large-scale binary and categorical data.
method Federated variational inference with local merge and delete moves in parallel batches, followed by global merge moves.
result Empirical validation shows superior performance compared to existing algorithms.
UPCA solves data matrix completion with permuted columns.
problem Data matrix completion with permuted columns.
method Algebraic geometry and two-stage algorithm.
result UPCA recovers the ground-truth matrix from corrupted data.
Neural circuits contain heterogeneous groups of neurons that differ in type, location, connectivity, and basic response properties. However, traditional methods for dimensionality reduction and clustering are ill-suited to recovering the structure underlying the organization of neural circuits. In particular, they do n…