# Speech-Generation **Repository Path**: zebiak/Speech-Generation ## Basic Information - **Project Name**: Speech-Generation - **Description**: Textual Speech generation using LSTM network - **Primary Language**: Unknown - **License**: Not specified - **Default Branch**: master - **Homepage**: None - **GVP Project**: No ## Statistics - **Stars**: 0 - **Forks**: 0 - **Created**: 2020-07-16 - **Last Updated**: 2021-06-20 ## Categories & Tags **Categories**: Uncategorized **Tags**: None ## README # Speech-Generation Textual Speech generation using LSTM network We test two popular architecture for language modelling and test for word-prediction task. * Vanilla LSTM with 100 context words * Bidirectional LSTM with 10 context words ## 1. Dataset Dataset used is the 'Corpus of Presidential Speeches' by Grammer Lab. Link: [Dataset](http://www.thegrammarlab.com/?nor-portfolio=corpus-of-presidential-speeches-cops-and-a-clintontrump-corpus#) The dataset consist of 43 sets of presidential speeches for 43 different Presidents of the USA. 1. coolidge's 12 speeches 2. tyler's 18 speeches 3. wilson's 32 speeches 4. ford's 14 speeches 5. pierce's 15 speeches 6. lincoln's 15 speeches * * * * 41. grant's 32 speeches 42. jqadams's 8 speeches 43. jackson's 26 speeches ### 1.1. Data Collection We use Lyndon B. Johnson's speeches as it has 71 speeches (with 71 individual .txt files). We concatenated these 71 files into a single .txt file. This summed up to a 2.82M words and a vocabulary size of 8703. Developing a Speech Generator on this requires huge amount of memory. Google Colaboratory provides 12GB VRAM on Google's NVIDIA K80 powered GPU runtime. Output vector which would be of shape: (2.82M, >8703) is not suitable for language modelling tasks. Filtering words and inflection bring down the size to (1.34M, 8703). Furthermore, after data pre-processing the Vanilla LSTM Model has a bottleneck at the Softmax Layer (Output Layer) due its size: O(|V|) and slows down training. This is addressed using different variation of probability estimation method. Our Project: [Hierarchical Softmax](https://github.com/AshwinDeshpande96/Hierarchical-Softmax) addresses this problem. ### 1.2. Data Pre-Processing The text files contain several punctuation-symbols, numbers, spacings and word inflection. It is important to be careful and try to remove characters or letters such that it helps reduce the vocabulary size. Otherwise if vocabulary is large the numbers of classes increases. And the output layer will now have too many classes to predict. Large number of classes will slow down training and will require large resources and time to converge. text = "Today's weather condition is cloudy with a 76% of rain. Temperature may remain cool at 21°C with Humidity 61%. Rainfall so far is measured at 130mm." #### 1.2.1. Following punctuations have been removed: " ' . & , ? / : ; < > $ # @ ! % * ( ) [ ] { } \n - tokenizer = RegexpTokenizer(r'\w+') words = tokenizer.tokenize(text) Output: text = "Todays weather condition is cloudy with a 76 of rain Temperature may remain cool at 21 C with Humidity 61 Rainfall so far is measured at 130mm " #### 1.2.2. Word Inflections are reduced to its root words: We use Porter Stemming or WordNet Lemmatization we brought down Vocabulary to 6673 or 8703 respectively. Output: text = "today weath condit is cloudy with a seventy six of rain temp may remain cool at twenty on c with humid sixt on rainfal so far is meas at on hundr and thirty mm " These words without their inflection make little sense, hence a dictionary variable is used where each root word entry is mapped to it's original inflection: dict = { today: Today's, weath: weather, condit: condition, is: is, cloudy: cloudy, with: with, . . . hundr: hundred, and: and, thirty: thirty, mm: mm } This however creates an issue: dict will have only one entry for a root word of different inflections. This is substantially larger than it's character wise prediction counterpart. In comparison, the vocabulary for character prediction in worst case is as large as the ASCII character set: 256 characters. For natural language however vocabulary will consist of '[a-z, A-Z]' and punctuation. Prediction using character doesn't reflect human intelligence of forming sentences. Humans learn to form sentences by recognizing a word's contextual importance i.e. humans can infer a word's meaning depending on it's pattern of occurence. n-gram models mimic this notion and calculate probabilities of next word succeeding a sequence based on it's prior occurence count. But, to predict a word wi, preceded by a sequence of contextual words: [wi-1,wi-2 . . . w1], the probability may not always be contingent to it's prior occurence count. This approach has several disadvantages: 1. Predicted wi may only occur as a part of word pair. Given a dataset of news-media collected from San-Franisco Bay Area. It is natural that the word Fransisco occurs frequently in this text. Hence the probability P(wi='Fransisco') is very likely. Even though 'Fransisco' will almost never occur without 'San', P(wi='Fransisco') will be most probable even if wi-1 is not 'San'. Vice-versa predicted P(wi='Fransisco') could be not likely if its frequency is low in a dataset, even if wi-1 is 'San'. 2. When wi does not occur in dataset but is the correct word after wi-1 (It's first occurence is encountered in the test set) Predicted wi will be one of |V| words with highest probability, even though wi was never encountered in training set. To counter this problem, some probability mass from vocabulary is subtracted and assigned to the new word. But, how much probability mass is to be given to new word? This cannot be estimated deterministically. However, several approaches such as back-off, interpolation and discounting techniques are proposed to mitigate these problems. Some of the popular ones are: * Stupid back-off * Katz's back-off model * Good–Turing discounting * Witten–Bell discounting * Lidstone's smoothing * Kneser–Ney smoothing For more details on n-gram language models: * [INVESTIGATION OF BACK-OFF BASED INTERPOLATION BETWEEN RECURRENT NEURAL NETWORK AND N-GRAM LANGUAGE MODELS](http://mi.eng.cam.ac.uk/~mjfg/asru15-chen.pdf) * [Stanford NLP](https://www.youtube.com/watch?v=Saq1QagC8KY&list=PLQiyVNMpDLKnZYBTUOlSI9mi9wAErFtFm&index=12) We can see that deterministic approach such as n-gram do not take into account the contextual information in dataset. It is not possible to manually design an algorithm to recognize occurence patterns. In order to build an algorithm to identify such patterns we need to understand what the logic is supposed to look for. Neither do we have a method to validate if found pattern is correct. Recurrent Neural Network are best suited for this task. We treat the neural network as a black box, wherein said patterns are recognized such that sequential context data is taken into account. RNNs still fail to take into account the long-distance context dependencies, as they are sparsely distributed throughout the data. Learn more about it here: [Yoshua Bengio - Presentation](https://www.bilibili.com/video/av34864474/), [Shortcomings of regular Encoder-Decoder Model(without Attention)](https://machinelearningmastery.com/encoder-decoder-attention-sequence-to-sequence-prediction-keras/) ## 2. Model * Vanilla LSTM network defined as follows: [Vanilla LSTM Network](https://github.com/AshwinDeshpande96/Speech-Generation/blob/master/president_NLP.ipynb)

Fig-1: Vanilla LSTM (Network Definition)

LSTM's ability to estimate sequential patterns has many applications such as: * Gene Classification * Speech Synthesis * Music Generation * Machine Translation We can see one such implementation here: [Music Generation](https://towardsdatascience.com/how-to-generate-music-using-a-lstm-neural-network-in-keras-68786834d4c5) Network proposed here is suitable for vocabulary of smaller size. Even though the maximum computation workload is seen at Dense Layer and further Softmax scoring, networks with deeper LSTMs incur larger training time. Hence we use a network of single LSTM layer. This takes several days to train. Next-Word prediction is not subject to overfitting. Hence, we use the entire dataset for training and none for validation. Because our goal here is to fit a model that behaves exactly like the dataset, out model is a high variance-low bias model. We only minimize the training error without validating across unseen data (Validation Set). * Bidirectional LSTM network defined as follows: [biLSTM Network](https://github.com/AshwinDeshpande96/Speech-Generation/blob/master/biLM.ipynb) ![Image of biLSTM](https://github.com/AshwinDeshpande96/Speech-Generation/blob/master/bidirection_network.png)

Fig-2: Bidirectional LSTM (Network Definition)

This type of architecture produces a contextual representation of a word, which is extrememly important in NLP tasks such as: * Word Sense Disambiguation * Parts of Speech Tagging * Named Entity Recognition * Coreference Resolution Furthermore learned embedding feature vectors form input for every other NLP tasks: * Question Answering * Textual Entailment * Semantic Role Labelling * Sentiment Analysis In this example (Figure-3), it is shown how each word (here: 'her' and 'have') is represented by a feature vector generated by the network. By reading the sentence (Figure-3), we can infer that word 'her' correponds to 'Hillary' (to the left of the word), while word 'those' corresponds to 'working with her' (to the right of the word). Based on these references, we are able to answer who 'her' and 'those' are depending on where in the sequence it occured. Thus it is beneficial to have word representations that are defined based on it's context. "Over the last 8 months, Hillary and those working with her have talked to literally thousands of Americans to understand the strengths and the frailties of this system of ours. They met with over 1,100 health care organizations." ![Image of biLSTM](https://github.com/AshwinDeshpande96/Speech-Generation/blob/master/biLM.jpeg)

Fig-3: Bidirectional LSTM (Network Architecture)

In NLP tasks such as Word Sense Disambiguation it is necessary to identify the meaning of a word. As in English language words can have multiple meanings (Polysemy), this is a difficult task in absence of word context. For example take the word 'have' can mean: * possess, own, or hold. * experience; undergo. * be obliged or find it necessary to do the specified thing. * perform the action indicated by the noun specified. * show (a personal attribute or quality) by one's actions or attitude. * place or keep (something) in a particular position. * be the recipient of (something sent, given, or done). In our text it can mean 'perform the action indicated by the noun specified' which cannot be identified by n-grams models. Further we can see the benefits of having bidirectional model as the references maybe defined in either of the directions. Since context sizes are fixed it cannot be said that every important context information is present within given context. Hence, it is necessary to increase the size of the context such that possible references are within that context range of the word. This however has a disadvantage: we cannot definitively correlate the word and it's reference as it's positions are unknown, this is due to the sparsity of the contextual data. This problem is solved using the concept of 'Attention'. This is however, not explored in this project, but you can read about it here: [Neural Machine Translation by Jointly Learning to Align and Translate](https://arxiv.org/abs/1409.0473) and [Andrew NG's - Attention Model Intuition](https://www.youtube.com/watch?v=SysgYptB198) ## 3. Results Since RNNs are a highly sequential type of network, it is not very parallelizable. This property incurs huge training times. Our network was trained for approximately 7 days.