Information Extraction — Tree Alignment Approach to Pattern Discovery in Web Documents

Abstract

The World Wide Web has nowen tered its mature age. It not only hosts and serves large amounts of pages but also offers large amounts of information potentially useful for individuals and businesses. Modern decision support can no more be effective without timely and accurate access to this unprecedented source of data. However, unlike in a database, the structure of data available on the Web is not known apriori and its understanding seems to require human intervention. Yet the conjunction of layout rules and simple domain knowledge enables in many cases the automatic understanding of such unstructured data. In such cases we say that data is semi-structured. Wrapper generation for automatic extraction of information from theWeb has therefore been a crucial challenge in the recent years. Various authors have suggested different approaches for extracting semi-structured data from the Web, ranging from analyzing the layout and syntax of Web documents to learning extraction rules from user’s training examples. In this paper, we propose to exploit the HTML structure of Web documents that contain information in the form of multiple homogeneous records. We use a Tree Alignment algorithm with a novel combination of heuristics to detect repeated patterns and infer extraction rules. The performance study shows that our approach is effective in practice, yielding practical performance and accurate results.