Abstract
Natural Language Generation (NLG) is one of the most critical yet challenging tasks in all Natural Language Processing applications. It is a process to automate text generation so that humans can understand its meaning. A handful of research articles published in the literature have described how NLG can produce understandable texts in various languages. The use of sequence-to-sequence modeling powered by deep learning techniques such as Long Term Short Term Memory, Recurrent Neural Networks, and Gated Recurrent Units has received much popularity as text generators. This survey provides a comprehensive overview of text generations and their related techniques, such as statistical, traditional, and neural network-based techniques. Generating text using the sequence-to-sequence model is not a simple task as it needs to handle continuous data, such as images, and discrete information, such as text. Therefore, in this study, we have identified some crucial areas for further research on text generation, such as incorporating a large text dataset, identifying and resolving grammatical errors, and generating extensive sentences or paragraphs. This work has also presented a detailed overview of the activation functions used in deep learning-based models and the evaluation metrics used for text generation.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Abbreviations
- AI:
-
Artificial intelligence
- ML:
-
Machine learning
- NLP:
-
Natural language processing
- NLG:
-
Natural language generation
- NLU:
-
Natural language understanding
- RNN:
-
Recurrent neural network
- LSTM:
-
Long short term memory
- GRU:
-
Gated recurrent unit
- CNN:
-
Convolution neural network
- POS:
-
Part of speech
- NER:
-
Named entity recognition
- CBOW:
-
Common bag of word
- GloVe:
-
Global vector
- RST:
-
Rhetorical structure theory
- REG:
-
Referring expression generation
- BRNN:
-
Bi-direction recurrent neural network
- ReLU:
-
Rectified linear activation function
- ROUGE:
-
Recall Oriented Understudy for Gisting Evaluation
- BLEU:
-
Bilingual evaluation understudy
- TCN:
-
Temporal convolution nets
- MCW:
-
Medical College of Wisconsin
- LCS:
-
Longest common subsequence
References
Dethlefs N, Schoene A, Cuayáhuitl H (2021) A divide-and-conquer approach to neural natural language generation from structured data. Neurocomputing 433:300–309. https://doi.org/10.1016/j.neucom.2020.12.083
Cao J (2020) Generating natural language descriptions from tables. IEEE Access 8:46206–46216. https://doi.org/10.1109/ACCESS.2020.2979115
Wolf T et al (2020) Transformers: state-of-the-art natural language processing, pp 38–45
Ruder S (2019) Neural transfer learning for natural language processing
Song M (2021) A study on the predictive analytics powered by the artificial intelligence in the movie industry. Int J Adv smart Converg 10(4):72–83
Weizenbaum J (1983) ELIZA—a computer program for the study of natural language communication between man and machine. Commun ACM 26(1):23–28. https://doi.org/10.1145/357980.357991
Colby KM (1976) Artificial paranoia: a computer simulation of paranoid processes, vol 7, no 1
Angeli G, Liang P, Klein D (2010) A simple domain-independent probabilistic approach to generation. In: EMNLP 2010—conference on empirical methods in natural language processing, proceedings of the conference, pp 502–512
Meister C, Pimentel T, Wiher G, Cotterell R (2022) Typical decoding for natural language generation. 2022, [Online]. Available: http://arxiv.org/abs/2202.00666
McShane M, Leon I (2022) Language generation for broad-coverage, explainable cognitive systems. Adv Cogn Syst X, pp 1–6 [Online]. Available: https://arxiv.org/abs/2201.10422v1
Li Z (2022) Text language classification based on dynamic word vector and attention mechanism. In: 2021 international conference on big data analytics for cyber-physical system in smart city, pp 367–375
Elahi GMM, Yang YH (2022) Online learnable keyframe extraction in videos and its application with semantic word vector in action recognition. Pattern Recognit. https://doi.org/10.1016/j.patcog.2021.108273
Pennington J, Socher R, Manning C (2014) {G}lo{V}e: global vectors for word representation. In: Proceedings of the 2014 conference on empirical methods in natural language processing ({EMNLP}), pp 1532–1543. https://doi.org/10.3115/v1/D14-1162
Dharma EM, Gaol FL, Warnars HLHS, Soewito B (2022) The accuracy comparison among Word2Vec, glove, and fasttext towards convolution neural network (CNN) text classification. J Theor Appl Inf Technol 100(2):349–359
Nandanwar AK, Choudhary J (2021) Semantic features with contextual knowledge-based web page categorization using the glove model and stacked bilstm. Symmetry (Basel). https://doi.org/10.3390/sym13101772
Jagfeld G, Jenne S, Vu NT (2018) Sequence-to-sequence models for data-to-text natural language generation: word- vs. character-based processing and output diversity. In: INLG 2018—11th International Natural Language Generation Conference, Proceedings, pp 221–232. https://doi.org/10.18653/v1/w18-6529
Gaur M, Arora M, Prakash V, Kumar Y, Gupta K, Nagrath P (2022) Analyzing natural language essay generator models using long short-term memory neural networks, pp 233–248
Kannan S, Vathsala MK (2022) Mathematical model for application of natural language description in the creation of an animation. In: Emerging research in computing, information, communication and applications, pp 237–251
Shi J, Yang Z, He J, Xu B, Lo D (2022) Can Identifier Splitting Improve Open-Vocabulary Language Model of Code?, no. 1, [Online]. Available: http://arxiv.org/abs/2201.01988
Li M et al (2022) Automated data function extraction from textual requirements by leveraging semi-supervised CRF and language model. Inf Softw Technol 143:106770. https://doi.org/10.1016/j.infsof.2021.106770
Liu Y, Wang L, Shi T, Li J (2021) Detection of spam reviews through a hierarchical attention architecture with N-gram CNN and Bi-LSTM. Inf Syst 103:101865. https://doi.org/10.1016/j.is.2021.101865
Lin J, Sun G, Beydoun G, Li L (2022) Applying machine translation and language modelling strategies for the recommendation task of micro learning service. Educ Technol Soc 25(1):205–212
Reiter E, Dale R (1997) Building applied natural language generation systems. Nat Lang Eng 3(1):57–87. https://doi.org/10.1017/S1351324997001502
Kunhi LM, Shetty J (2022) Generation of structured query language from natural language using recurrent neural networks. Invent Commun Comput Technol 63–73
Zhang X, Lapata M (2014) Chinese poetry generation with recurrent neural networks. In: Proceedings of the 2014 conference on empirical methods in natural language processing ({EMNLP}), pp 670–680. https://doi.org/10.3115/v1/D14-1074
Gatt A, Krahmer E (2018) Survey of the state of the art in natural language generation: core tasks, applications and evaluation. J Artif Intell Res 61(c):1–64. https://doi.org/10.1613/jair.5714
Palombella VJ, Rando OJ, Goldberg AL, Maniatis T (1994) The ubiquitin-proteasome pathway is required for processing the NF-kappa B1 precursor protein and the activation of NF-kappa B. Cell 78(5):773–785. https://doi.org/10.1016/s0092-8674(94)90482-0
Mann WC, Thompson SA (1987) Rhetorical structure theory: description and construction of text structures. In: Kempen G (ed) Natural language generation: new results in artificial intelligence, psychology and linguistics. Springer, Dordrecht, pp 85–95
Santhanam S (2020) Context based text-generation using LSTM networks. [Online]. Available: http://arxiv.org/abs/2005.00048
Langkilde I (2000) Forest-based statistical sentence generation. [Online]. Available: https://aclanthology.org/A00-2023
Yao T et al (2021) Compound figure separation of biomedical images with side loss. In: Deep generative models, and data augmentation, labelling, and imperfections: first workshop, DGM4MICCAI 2021, and first workshop, DALI 2021, held in conjunction with MICCAI 2021, Strasbourg, France, October 1, 2021, proceedings 1, pp 173–183
Iglesias P, Sicilia M-A, García-Barriocanal E (2023) Detecting browser drive-by exploits in images using deep learning. Electronics. https://doi.org/10.3390/electronics12030473
Zhao M et al (2021) VoxelEmbed: 3D instance segmentation and tracking with voxel embedding based deep learning. In: Machine learning in medical imaging, pp 437–446
Roy S, Viswanatham VM (2016) Classifying spam emails using artificial intelligent techniques. Int J Eng Res Africa 22:152–161. https://doi.org/10.4028/www.scientific.net/JERA.22.152
Roy S, Viswanatham VM, Krishna P (2016) Spam detection using hybrid model of rough set and decorate ensemble. Int J Comput Syst Eng 2:139. https://doi.org/10.1504/IJCSYSE.2016.079000
Wei M, Zhang Y (2019) Natural answer generation with attention over instances. IEEE Access 7:61008–61017. https://doi.org/10.1109/ACCESS.2019.2904337
Pawade D, Sakhapara A, Jain M, Jain N, Gada K (2018) Story scrambler—automatic text generation using word level RNN-LSTM. Int J Inf Technol Comput Sci 10(6):44–53. https://doi.org/10.5815/ijitcs.2018.06.05
Shen S, Chen Y, Yang C, Liu Z, Sun M (2018) Zero-shot cross-lingual neural headline generation. IEEE/ACM Trans Audio Speech Lang Process 26(12):2319–2327. https://doi.org/10.1109/TASLP.2018.2842432
Chen Y, Yang C, Liu Z, Sun M (2020) Reinforced zero-shot cross-lingual neural headline generation. IEEE/ACM Trans Audio Speech Lang Process 28(12):2572–2584. https://doi.org/10.1109/TASLP.2020.3009487
Abujar S, Masum AKM, Chowdhury SMMH, Hasan M, Hossain SA (2019) Bengali text generation using bi-directional RNN. In: 2019 10th International conference on computing and communication networks technology, ICCCNT 2019, pp 1–5. https://doi.org/10.1109/ICCCNT45670.2019.8944784
Bao J, Tang D, Duan N, Yan Z, Zhou M, Zhao T (2019) Text generation from tables. IEEE/ACM Trans Audio Speech Lang Process 27(2):311–320. https://doi.org/10.1109/TASLP.2018.2878381
Wang HC, Hsiao WC, Chang SH (2020) Automatic paper writing based on a RNN and the TextRank algorithm. Appl Soft Comput J 97:106767. https://doi.org/10.1016/j.asoc.2020.106767
Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735
Roy S, Kaul D, Roy R, Barna C, Mehta S, Misra A (2018) Prediction of customer satisfaction using Naive Bayes, multiclass classifier, K-star and IBK
Ren Y, Hu W, Wang Z, Zhang X, Wang Y, Wang X (2021) A hybrid deep generative neural model for financial report generation. Knowl Based Syst 227:107093. https://doi.org/10.1016/j.knosys.2021.107093
Hoogi A, Mishra A, Gimenez F, Dong J, Rubin D (2020) Mammography reports simulation, vol 24, no 9, pp 2711–2717
Xiang L, Yang S, Liu Y, Li Q, Zhu C (2020) Novel linguistic steganography based on character-level text generation. Mathematics 8(9):1–18. https://doi.org/10.3390/math8091558
Chakraborty S, Banik J, Addhya S, Chatterjee D (2020) Study of dependency on number of LSTM units for character based text generation models. In: 2020 International conference on computer science and engineering and applications, ICCSEA 2020. https://doi.org/10.1109/ICCSEA49143.2020.9132839
Sanzidul IM, Sadia Sultana SM, Abujar S, Hossain SA (2019) Sequence-to-sequence Bangla sentence generation with LSTM recurrent neural networks. Procedia Comput Sci 152:51–58. https://doi.org/10.1016/j.procs.2019.05.026
Liu T, Wang K, Sha L, Chang B, Sui Z (2018) Table-to-text generation by structure-aware seq2seq learning. In: 32nd AAAI conference on artificial intelligence, AAAI 2018, pp 4881–4888
Sha L et al (2018) Order-planning neural text generation from structured data. In: 32nd AAAI conference on artificial intelligence, AAAI 2018, pp 5414–5421
Fan A, Lewis M, Dauphin Y (2018) Hierarchical neural story generation. In: ACL 2018—56th annual meeting of the association for computational linguistics, proceedings conference (long papers), vol 1, pp 889–898. https://doi.org/10.18653/v1/p18-1082
Li J, Monroe W, A Ritter, Galley M, Gao J, Jurafsky D (2016) Deep reinforcement learning for dialogue generation. IN: EMNLP 2016—conference on empirical methods in natural language processing proceedings, no 4, pp 1192–1202. https://doi.org/10.18653/v1/d16-1127
Bourane S et al (2015) Gate control of mechanical itch by a subpopulation of spinal cord interneurons. Science 350(6260):550–554. https://doi.org/10.1126/science.aac8653
Zheng Q, Yang M, Yang J, Zhang Q, Zhang X (2018) Improvement of generalization ability of deep CNN via implicit regularization in two-stage training process. IEEE Access 6:15844–15869. https://doi.org/10.1109/ACCESS.2018.2810849
Zhu J, Li J, Zhu M, Qian L, Zhang M, Zhou G (2020) Modeling graph structure in transformer for better AMR-to-text generation. In: EMNLP-IJCNLP 2019—2019 conference on empirical methods natural language processing, 9th international joint conference natural language processing proceedings, vol 1, pp 5459–5468. https://doi.org/10.18653/v1/d19-1548
Biswas R, Vasan A, Roy SS (2020) Dilated deep neural network for segmentation of retinal blood vessels in fundus images. Iran J Sci Technol Trans Electr Eng 44(1):505–518. https://doi.org/10.1007/s40998-019-00213-7
Schmitt M, Sharifzadeh S, Tresp V, Schütze H (2020) An unsupervised joint system for text generation from knowledge graphs and semantic parsing. In EMNLP 2020—2020 conference on empirical methods natural language processing proceedings, pp 7117–7130. https://doi.org/10.18653/v1/2020.emnlp-main.577
Qader R, Jneid K, Portet F, Labbé C (2018) Generation of company descriptions using concept-to-text and text-to-text deep models: dataset collection and systems evaluation. In: Proceedings of the 11th international conference on natural language generation, pp 254–263. https://doi.org/10.18653/v1/W18-6532
Jin D, Jin Z, Hu Z, Vechtomova O, Mihalcea R (2022) Deep learning for text style transfer: a survey. Comput Linguist 48(1):155–205. https://doi.org/10.1162/COLI_a_00426
Yermakov R, Ag B, Drago N, Ag B, Ziletti A, Ag B (2021) Biomedical data-to-text generation via fine-tuning transformers, pp 364–370
Kim Y, Jang M, Allan J (2020) Explaining text matching on neural natural language inference. ACM Trans Inf Syst 38(4):1–23
Wang M, Lu S, Zhu D, Lin J, Wang Z (2018) A high-speed and low-complexity architecture for softmax function in deep learning. In: 2018 IEEE Asia Pacific conference on circuits and systems (APCCAS), pp 223–226. https://doi.org/10.1109/APCCAS.2018.8605654
Bouchard G (2007) Efficient bounds for the softmax function, applications to inference in hybrid models. Nips 1–9 [Online]. Available: http://eprints.pascal-network.org/archive/00003498/
Yin X, Goudriaan J, Lantinga EA, Vos J, Spiertz HJ (2003) A flexible sigmoid function of determinate growth. Ann Bot 91(3):361–371. https://doi.org/10.1093/aob/mcg029
Lin C-Y (2004) {ROUGE}: a package for automatic evaluation of summaries. In: Text summarization branches out, pp 74–81. Available: https://aclanthology.org/W04-1013
Lin C-Y (2004) Looking for a few good metrics: ROUGE and its evaluation. In: NTCIR Work, pp 1–8
Yadav D et al (2022) Qualitative analysis of text summarization techniques and its applications in health domain. Comput Intell Neurosci 2022:1–14. https://doi.org/10.1155/2022/3411881
Yadav AK et al (2022) Extractive text summarization using deep learning approach. Int J Inf Technol. https://doi.org/10.1007/s41870-022-00863-7
Sun Y et al (2022) Bidirectional difference locating and semantic consistency reasoning for change captioning. Int J Intell Syst. https://doi.org/10.1002/int.22821
Papineni K, Roukos S, Ward T, Zhu WJ (2002) BLEU: a method for automatic evaluation of machine translation. https://doi.org/10.3115/1073083.1073135
Singh C (2017) Alice in Wonderland Gutenberg. https://www.kaggle.com/datasets/chandan2495/alice-in-wonderland-gutenbergproject/metadata
BG illustrated by A. Browne, Hansel and Gretel (1981). Julia MacRae Books, London, New York
Author information
Authors and Affiliations
Contributions
Conceptualization, AKP, and SSR; methodology, AKP; software, AKP; validation, AKP, and SSR; formal analysis, AKP; investigation, AKP; resources, AKP; data curation, AKP; writing—original draft preparation, AKP; writing—review and editing, AKP and SSR; visualization, AKP; supervision, SSR; All authors have read and agreed to the published version of the manuscript.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Pandey, A.K., Roy, S.S. Natural Language Generation Using Sequential Models: A Survey. Neural Process Lett 55, 7709–7742 (2023). https://doi.org/10.1007/s11063-023-11281-6
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11063-023-11281-6