Abstract
Recently, with the rapid developments of the Internet and social networks, there have been tremendous increase in the amount of complex-structured text resources. These information explosions require extensive studies as well as more advanced methods in order to better understand and effectively model/learn these high-dimensional/structure-complicated textual datasets. Moving along with the recent progresses in deep learning and textual representation learning approaches, many researchers in this domain have been attracted by utilizing different deep neural architectures for learning essential features from texts. These novel neural architectures must enable to handle complex textual feature engineering. Moreover, it also has to be able to extract deeper semantic and structural information from textual resources. Recently, there are several integrations between advanced deep learning architectures, such as recurrent neural networks (RNNs), sequence-to-sequence (seq2seq) and transformers in text classification have been proposed. These hybrid deep neural architectures have shed light on how computers can comprehensively process sequential information from texts to fine-tune for leveraging the performance of multiple tasks in natural language processing, including classification. However, most of recent RNN-based techniques still suffer from several limitations. These limitations are mainly related to the capability of capturing the global long-range dependent as well syntactical structures of the given text corpus. There are some recent studies have shown that a combination of graph-based text representation and graph neural network (GNN) approaches can cope with these challenges. In this survey works, we mainly focus on discussing about recent state-of-the-art studies which are mainly dedicated on the text graph representation learning through GNN, named as TG-GNN. In addition, beside the TG-GNN based models’ features and capability discussions, we also mentioned about the pros/cons. Extensive comparative studies of TG-GNN based techniques in benchmark datasets for text classification problem are also provided in this survey. Finally, we highlight existing challenges as well as identify perspectives which might be useful for future improvements in this research direction.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Notes
Pre-trained Word2Vec model: https://code.google.com/archive/p/word2vec/
Pre-trained GloVe model: https://nlp.stanford.edu/projects/glove/
Pre-trained fastText model: https://fasttext.cc/docs/en/english-vectors.html
Pre-trained BERT model: https://github.com/google-research/bert
TextGCN (python): https://github.com/codeKgu/Text-GCN
TensorGCN (python): https://github.com/THUMLP/TensorGCN
TextING (python): https://github.com/CRIPAC-DIG/TextING
20-NewsGroups dataset: http://qwone.com/~jason/20Newsgroups/
WebKB dataset: http://www.cs.cmu.edu/~webkb/
Ohsumed dataset: http://disi.unitn.it/moschitti/corpora.htm
Movie Reviews (MR) dataset: https://github.com/mnqu/PTE/tree/master/data/mr
Amazon Reviews (AR) dataset: https://snap.stanford.edu/data/web-Amazon.html
IMDb dataset: https://datasets.imdbws.com/
TextRNN (Python) (Hemmatian and Sohrabi 2019): https://github.com/ShawnyXiao/TextClassification-Keras
CNN-based text classification (Python) (Singh et al. 2021): https://github.com/yoonkim/CNN_sentence
Abbreviations
- AE:
-
Auto-encoding/auto-encoder
- CNN:
-
Convolutional neural network
- DL:
-
Deep learning
- GAT:
-
Graph attention network
- GCN:
-
Graph convolutional network
- GNN:
-
Graph neural network
- LSTM:
-
Long-short term memory
- MLP:
-
Multi-layer perception
- NLP:
-
Natural language processing
- TG-GNN:
-
Text graph representation learning through graph neural network
- \(\mathcal{d}\) and \(\mathcal{D}\) :
-
A document and a text corpus, respectively
- \(\mathcal{w}\) and \(\mathcal{W}\) :
-
A word and a vocabulary set of the given text corpus, respectively
- \(\mathcal{G}=(\mathcal{V},\mathcal{E})\) :
-
A graph-based structure with a set of nodes (\(\mathcal{V},\mathrm{v}\in \mathcal{V}\)) and edges (\(\mathcal{E},\mathrm{e}\in \mathcal{E}\))
- \(\mathcal{A}\) :
-
The adjacency matrix of a given graph
- \(\widehat{\mathcal{A}}\) :
-
Normalized version of adjacency matrix
- \(\mathcal{H}\) :
-
The hidden state of a given neural network architecture
- \(\upsigma (.)\) :
-
The sigmoid activation function
- \(\mathrm{ReLU}(.)\) :
-
The rectified linear unit activation function
- \(\mathrm{softmax}(.)\) :
-
The softmax function
- \(\mathcal{Y}\) and \(\widehat{\mathcal{Y}}\) :
-
The sets of classification ground-truth and prediction labels, respectively
References
Altınel B, Ganiz MC (2018) Semantic text classification: a survey of past and recent advances. Inf Process Manage 54(6):1129–1153
Chen F, Wang YC, Wang B, Kuo CCJ (2020) Graph representation learning: a survey. APSIPA Trans Signal Inf Process 9:e15
Dhar A, Mukherjee H, Dash NS, Roy K (2021) Text categorization: past and present. Artif Intell Rev 54(4):3007–3054
Hemmatian F, Sohrabi MK (2019) A survey on classification techniques for opinion mining and sentiment analysis. Artif Intell Rev 52(3):1495–1545
Hu L, Li C, Shi C, Yang C, Shao C (2020a) Graph neural news recommendation with long-term and short-term interest modeling. Inf Process Manage 57(2):102142
Kadhim AI (2019) Survey on supervised machine learning techniques for automatic text classification. Artif Intell Rev 52(1):273–292
Kowsari K, Jafari Meimandi K, Heidarysafa M, Mendu S, Barnes L, Brown D (2019) Text classification algorithms: a survey. Inf 10(4):150
Ma G, Ahmed NK, Willke TL, Philip SY (2021) Deep graph similarity learning: a survey. Data Min Knowl Discov 35:688
Minaee S, Kalchbrenner N, Cambria E, Nikzad N, Chenaghlu M, Gao J (2021) Deep learning–based text classification: a comprehensive review. ACM Comput Surv (CSUR) 54(3):1–40
Murari A, Gelfusa M, Lungaroni M, Gaudio P, Peluso E (2021) A systemic approach to classification for knowledge discovery with applications to the identification of boundary equations in complex systems. Artif Intell Rev 55:255
Nguyen G, Dlugolinsky S, Bobák M, Tran V, García ÁL, Heredia I et al (2019) Machine learning and deep learning frameworks and libraries for large-scale data mining: a survey. Artif Intell Rev 52(1):77–124
Pham P, Nguyen LT, Vo B, Yun U (2021) Bot2Vec: a general approach of intra-community oriented representation learning for bot detection in different types of social networks. Inf Syst 103:101771
Pham P, Nguyen LT, Nguyen NT, Pedrycz W, Yun U, Vo B (2022a) ComGCN: community-driven graph convolutional network for link prediction in dynamic networks. IEEE Trans Syst Man Cybern Syst. https://doi.org/10.1109/TSMC.2021.3130149
Pham P, Pedrycz W, Vo B (2022b) Dual attention-based sequential auto-encoder for Covid-19 outbreak forecasting: a case study in Vietnam. Expert Syst Appl 203:117514
Pintas JT, Fernandes LA, Garcia ACB (2021) Feature selection methods for text classification: a systematic literature review. Artif Intell Rev 54:6149
Qiu X, Sun T, Xu Y, Shao Y, Dai N, Huang X (2020) Pre-trained models for natural language processing: a survey. Sci China Technol Sci 63:1872
Shivakumara P, Alaei A, Pal U (2021) Mining text from natural scene and video images: a survey. Wiley Interdiscip Rev Data Min Knowl Discov 11:e1428
Singh RK, Sachan MK, Patel RB (2021) 360 degree view of cross-domain opinion classification: a survey. Artif Intell Rev 54(2):1385–1506
Subhashini LDCS, Li Y, Zhang J, Atukorale AS, Wu Y (2021) Mining and classifying customer reviews: a survey. Artif Intell Rev 54:6343
Wu Z, Pan S, Chen F, Long G, Zhang C, Philip SY (2020) A comprehensive survey on graph neural networks. IEEE Trans Neural Netw Learn Syst 32(1):4–24
Wu JMT, Li Z, Herencsar N, Vo B, Lin JCW (2021) A graph-based CNN-LSTM stock price prediction algorithm with leading indicators. Multimedia Syst. https://doi.org/10.1007/s00530-021-00758-w
Xu S, Xiang Y (2021) Frog-GNN: multi-perspective aggregation based graph neural network for few-shot text classification. Expert Syst Appl 176:114795
Zhou J, Cui G, Hu S, Zhang Z, Yang C, Liu Z et al (2020) Graph neural networks: a review of methods and applications. AI Open 1:57–81
Zucco C, Calabrese B, Agapito G, Guzzi PH, Cannataro M (2020) Sentiment analysis for mining texts and social networks data: methods and tools. Wiley Interdiscip Rev Data Min Knowl Discov 10(1):e1333
Bahdanau D, Cho K, Bengio Y (2015) Neural machine translation by jointly learning to align and translate. In: ICLR
Blunsom P, Grefenstette E, Kalchbrenner N (2014) A convolutional neural network for modelling sentences. In: ACL
Conneau A, Schwenk H, Cun YL, Barrault L (2017) Very deep convolutional networks for text classification. In: ACL
Devlin J, Chang MW, Lee K, Toutanova K (2019) Bert: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Ding K, Wang J, Li J, Li D, Liu H (2020) Be more with less: hypergraph attention networks for inductive text classification. In: EMNLP
Fei H, Tan S, Li P (2019) Hierarchical multi-task word embedding learning for synonym prediction. In: SIGKDD
Hamilton WL, Ying R, Leskovec J (2017) Inductive representation learning on large graphs. In: NIPS
Howard J, Ruder S (2018) Universal language model fine-tuning for text classification. https://arXiv.org/1801.06146
Hu L, Xu S, Li C, Yang C, Shi C, Duan N et al (2020b) Graph neural news recommendation with unsupervised preference disentanglement. In: ACL
Huang L, Ma D, Li S, Zhang X, Wang H (2019) Text Level Graph Neural Network for Text Classification. In: EMNLP-IJCNLP
Kim Y (2014) Convolutional neural networks for sentence classification. In: EMNLP
Kipf TN, Welling M (2016) Semi-supervised classification with graph convolutional networks. In: ICLR
Le Q, Mikolov T (2014) Distributed representations of sentences and documents. In: International Conference on Machine Learning, PMLR
Li Q, Peng H, Li J, Xia C, Yang R, Sun L et al (2020) A survey on text classification: from shallow to deep learning. https://arXiv.org/2008.00364
Liu P, Qiu X, Chen X, Wu S, Huang XJ (2015) Multi-timescale long short-term memory neural network for modelling sentences and documents. In: EMNLP
Liu P, Qiu X, Huang X (2016) Recurrent neural network for text classification with multi-task learning. In: IJCAI
Liu X, You X, Zhang X, Wu J, Lv P (2020) Tensor graph convolutional networks for text classification. In: Proceedings of the AAAI Conference on Artificial Intelligence
Liu Y, Guan R, Giunchiglia F, Liang Y, Feng X (2021) Deep attention diffusion graph neural networks for text classification. In: EMNLP
Luong MT, Pham H, Manning CD (2015) Effective approaches to attention-based neural machine translation. In: EMNLP
Manning CD, Surdeanu M, Bauer J, Finkel JR, Bethard S, McClosky D (2014) The Stanford CoreNLP natural language processing toolkit. In: ACL
Mao R, Lin C, Guerin F (2018) Word embedding and wordnet based metaphor identification and interpretation. In: ACL
Mikolov T, Chen K, Corrado G, Dean J (2013) Efficient estimation of word representations in vector space. In: ICRL
Mikolov T, Grave É, Bojanowski P, Puhrsch C, Joulin A (2018) Advances in pre-training distributed word representations. In: Proceedings of the Eleventh International Conference on Language Resources and Evaluation
Nguyen D, Luo W, Nguyen TD, Venkatesh S, Phung D (2018a) Learning graph representation via frequent subgraphs. In: ICDM
Nguyen D, Luo W, Nguyen TD, Venkatesh S, Phung D (2018b) Sqn2vec: Learning sequence representation via sequential patterns with a gap constraint. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases
Pennington J, Socher R, Manning CD (2014) Glove: Global vectors for word representation. In: EMNLP
Peters ME, Neumann M, Iyyer M, Gardner M, Clark C, Lee K, Zettlemoyer L (2018) Deep contextualized word representations. https://arXiv.org/1802.05365
Radford A, Narasimhan K, Salimans T, Sutskever I (2018) Improving language understanding by generative pre-training. In: OpenAI
Rousseau F, Kiagias E, Vazirgiannis M (2015) Text categorization as a graph classification problem. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing
Sonawane SS, Kulkarni PA (2014) Graph based representation and analysis of text document: a survey of techniques. Int J Comput Appl 96(19)
Sutskever I, Vinyals O, Le QV (2014) Sequence to sequence learning with neural networks. In: NIPS
Tai KS, Socher R, Manning CD (2015) Improved semantic representations from tree-structured long short-term memory networks. In: ACL
Vaswani A et al (2017) Attention is all you need. In: NIPS
Veličković P, Cucurull G, Casanova A, Romero A, Lio P, Bengio Y (2018) Graph attention networks. In: ICLR
Vu T, Wang T, Munkhdalai T, Sordoni A, Trischler A, Mattarella-Micke A et al (2020) Exploring and predicting transferability across NLP tasks. In: EMNLP
Wang Z, Wang C, Zhang H, Duan Z, Zhou M, Chen B (2020) Learning dynamic hierarchical topic graph with graph convolutional network for document classification. In: International Conference on Artificial Intelligence and Statistics (PMLR)
Xie Q, Huang J, Du P, Peng M, Nie JY (2021) Inductive topic variational graph auto-encoder for text classification. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Yao L, Mao C, Luo Y (2019) Graph convolutional networks for text classification. In: Proceedings of the AAAI Conference on Artificial Intelligence
Zhang H, Zhang J (2020) Text graph transformer for document classification. In: EMNLP
Zhang Y, Yu X, Cui Z, Wu S, Wen Z, Wang L (2020) Every document owns its structure: inductive text classification via graph neural networks. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
Zhu X, Sobihani P, Guo H (2015) Long short-term memory over recursive structures. In: PMLR
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Pham, P., Nguyen, L.T.T., Pedrycz, W. et al. Deep learning, graph-based text representation and classification: a survey, perspectives and challenges. Artif Intell Rev 56, 4893–4927 (2023). https://doi.org/10.1007/s10462-022-10265-7
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10462-022-10265-7