Topic Modeling for Short Texts: A Novel Modeling Method

Abstract

Topic modeling is one of the major concerns in the short texts area, and mining these texts could uncover meaningful insights. However, the extreme short texts’ sparsity and imbalance bring new challenges to conventional topic models. In this paper, we combine a new ranking method with hierarchical representation for short text. Words ranking proves to be inexorable in generating value from disorganized short texts; thus, a novel ranking paradigm is developed and is referred to as the ordered biterm topic model (OBTM). OBTM models the semantic connection between every two words, regardless of whether they show up in similar short content, reinforcing the capacity to reveal the genuine semantic examples behind the corpus. The intense contextual information maintained in the space of word-to-word assures the word sensation in recognizing relevant topics with reliable quality. Then, the paradigm is associated with a hierarchical representation that captures the relations connecting the created topics. OBTM learns topics at the corpus level. This makes the inference more effective and robust, referring to hidden semantics patterns. Experiments on real-world collection reveal that OBTM can discover more relevant and coherent topics. It achieves high performance in various tasks and outperforming the state-of-the-art baselines.

Appendices

Appendix

Parameters Tuning

The manual tuning of the $\alpha , \beta$ hyper-parameters, is avoided, since the model figures out the exact values based on the statistical data distribution. In the preceding experiments, the model always achieves the best performances when $\alpha = 0.01$ and $\beta = 0.001$. The $\alpha$ is used as the prior hyper-parameters in the Dirichlet process to generate topics, while $\beta$ is used to generate words. The proposed approach determines the number of the hidden topic using a Dirichlet process, where the finite distribution of topics is sampled from a selected common base distribution, which considers the countably infinite set of possible topics. For all baseline models, Table 8 illustrates the experimental setting used in the original paper.

Table 8 The main common parameters setting for OBTM and the baselines methods

Evaluation Metrics

Short text topic models evaluation is an open problem, where a lot of metrics have been proposed for measuring the quality of the topics. To provide a good evaluation, this section highlights the evaluation metrics used in this paper.

1.1 Topic Coherence

Human topic ranking is the highest standard, and consequently a topic interpretability measure. In recent years, a new automatic evaluation methods are developed to evaluate the topic model quality. The topic coherence reflects the homogeneity for words, which contribute to the topic formulation. The proposed approach adopts the $C_V$ coherence measure proposed by Roder et al. [34]. Notably, the $C_V$ consists of four major parts. The first step is the data segmentation pairs, more formally, let $W = \{w_1, \ldots , w_N \}$ be the set of top-N words that describes a topic, then $S_i = \{(W', W^* )|W' = {w_i}; w_i \in W; W^* = W \}$, is the set of all pairs. For example, if $W = \{w_1, w_2, w_3\}$, then the pair $S_1 = \{(W' = w_1),(W^* = w_1, w_2, w_3)\}$. Douven et al. [7] assume that the segmentation measures the extent to which the subset $W^*$ supports or conversely undermines the subset $W'$.

The second step retrieves the probability of a single word $p(w_i)$, or the joint probability of two words $p(w_i, w_j )$, which can be guessed using their frequency over the corpus. The coherence measure $C_V$ creates a new virtual document using a frequency sliding window calculation. The window size creates a slid over the document by one-word token per step. The final probabilities $p(w_i)$ and $p(w_i, w_j )$ are calculated from the total number of virtual documents.

Given a pair $S_i = (W', W^* )$, the third step calculates a confirmation measure $\phi$, which reflects the strength of $W^*$ supports $W'$. Similarly to Aletras et al. [1], $W'$ and $W^*$ are represented as a means context vectors, that captures the semantic support of words in W using Eq. (10). Thus, the agreement between individual words $w_i$ and $w_j$ is calculated using Eq. (11) along normalized pointwise mutual information. Furthermore, the $\log$ operator is smoothed by adding $\epsilon$ to $\log p(w_i, w_j)$, and the $\gamma$ parameter controls the weight on higher NPMI values. $\phi$ is the confirmation measure for a given pair $S_i$ which is obtained by calculating the cosine vector similarity of all context vectors $\phi _{ S_i }(\vec {u}, \vec {w})$ (Eq. 12).

$$\begin{aligned} \mathbf {v}(W') = \left\{ \sum _{w_i \in W'} NMPI(w_i, w_j)^{\gamma } \right\} _{j=1,\ldots ,|W|} \end{aligned}$$

(10)

$$\begin{aligned} NMPI(w_i, w_j)^{\gamma } = \Bigg ( \frac{ \log \frac{p(w_i, w_j) + \epsilon }{p(w_i).p(w_j)}}{-\log {p(w_i, w_j) + \epsilon }} \Bigg )^{\gamma } \end{aligned}$$

(11)

$$\begin{aligned} \phi _{ S_i }(\vec {u}, \vec {w}) = \frac{\sum _{i=1} ^{|W|} {u_i \cdot w_i}}{\mid \mid \vec {u} \mid \mid _2 \cdot \mid \mid \vec {w} \mid \mid _2 } \end{aligned}$$

(12)

The final step returns the arithmetic mean of all confirmation measures $\phi$ as the final coherence score.

1.2 Pointwise Mutual Information

Since the $C_V$ metric evaluates the topic models quality internally, the proposed approach evaluated using the PMI-Score [25], which measures the topic coherence based on pointwise mutual information using external sources. Besides, these external data are model-independent, which makes the PMI-Score fair for all topic models. Given a topic k and its n probable words $(w_1, \ldots ,w_n)$, the PMI-Score measures the pairwise association between them is:

$$\begin{aligned} PMI_{Score}(k) = \frac{1}{n(1-n)} \sum _{1< i< j<n} PMI(w_i, w_j) \end{aligned}$$

(13)

$$\begin{aligned} PMI(w_i, w_j) = \log \frac{p(w_i, w_j) + \epsilon }{p(w_i)p(w_j)} \end{aligned}$$

(14)

This is an empirical conditional log-probability $\log p(w_j|w_i) = \log \frac{p(w_i, w_j)}{p(w_j)}$ smoothed by adding $\epsilon$ to $p(w_i, w_j)$.

1.3 Word Similarity

The conditional topic distribution for the word w can be defined as its semantic representation Eq. (15), where the $p(z_k|w)$ value is obtained after completing the Gibbs sampling step (Eq. 16).

$$\begin{aligned} s_w = [p(z_1|w), p(z_2|w), \ldots , p(z_k|w)], \end{aligned}$$

(15)

$$\begin{aligned} p(z_k|w) = \frac{n_{w|z_k}}{n_w} \end{aligned}$$

(16)

where $n_{w|z_k}$ stands for how many times w has be assigned with topic k during the sampling, and $n_w$ is the total occurrence of w in the given corpus. The distance between two words $w_i$ and $w_j$ represented by their semantic representations $s_i$ and $s_j$ is calculated using the Jensen–Shannon divergence:

$$\begin{aligned} JS(s_i,s_j) = \frac{1}{2} D_{KL} (s_j || m) +\frac{1}{2} D_{KL} (s_i || m) \end{aligned}$$

(17)

where $m = \frac{1}{2}(s_i+s_j)$ and $D_{KL} (p || q) = \sum _i p_i \ln \frac{p_i}{q_i}$ is the Kullback–Leibler divergence. The cosine similarity is also adopted to measure the distance between two words’ vectors, which is defined as:

$$\begin{aligned} Cosine(s_i,s_j) = \frac{s_i . s_j}{||s_i|| \quad ||s_j||} \end{aligned}$$

(18)