Supervised Negative Binomial Classifier for Probabilistic Record Linkage

Abstract

Motivated by the need for linking records across various databases, we propose a novel graphical model based classifier that uses a mixture of Poisson distributions with latent variables. The idea is to derive insight into each pair of hypothesis records that match by inferring its underlying latent rate of error using Bayesian Modeling techniques. The novel approach of using Gamma priors for learning the latent variables along with supervised labels is unique. The naive assumption is made deliberately as to the independence of the fields to propose a generalized theory for this class of problems and not to undermine the hierarchical dependencies that could be present in different scenarios. This classifier is able to work with sparse and streaming data. The application to record linkage is able to meet challenges of sparsity, data streams and varying nature of the datasets.