In&nbsp;Talking Data, we delve into the rapidly evolving worlds of Natural Language Processing and Generation. Text data is proliferating at a staggering rate, and only advanced coding languages like Python and R will be able to pull insights out of these datasets at scale. 



As the title suggests, in this article we’ll explore best practices in <a rel="noreferrer noopener" aria-label="natural language processing (opens in a new tab)" href="https://www.sisense.com/blog/r-vs-python-whats-the-best-language-for-natural-language-processing/" target="_blank">natural language processing</a> (NLP). To do this, we’ll use machine learning (ML) algorithms, which are based on concepts of linear algebra and statistics. These best practices will help us address a common question: “How do we represent text for machine learning systems?”



<h2 class="wp-block-heading">Text preprocessing</h2>



Understanding the real meaning of words by analyzing the context of the surrounding text is called semantic analysis. One of the first things you have to do for semantic analysis for an NLP project is text preprocessing. This is a vital practice in NLP and makes data more understandable for the algorithms. To understand text preprocessing, let&#8217;s use a common natural language processing task, sentiment analysis, as an example.



<a aria-label="Sentiment analysis (opens in a new tab)" href="https://www.sisense.com/blog/naive-bayes-sentiment-analysis-in-python-after-preparing-data-using-sql/" target="_blank" rel="noreferrer noopener">Sentiment analysis</a> has many applications, and it’s something we do as humans without really thinking. For instance: When customers write product reviews, a human reader can easily identify whether a given review is positive or negative. For computers, this process is much more complicated, so preprocessing steps are essential to get clean data.



Deep data exploration: Advanced analytics and insights using Python and R



<a class="action-btn " href="https://www.sisense.com/whitepapers/deep-data-exploration-advanced-analytics-and-insights-using-python-and-r/" target="_blank" rel="noopener noreferrer">Read more</a>



Let take a sample review posted for a product from https://www.fakereviewsite.com:&nbsp;



https://www.fakereviewsite.com Can&#8217;t do sugar.&nbsp; Have tried scores of SF Syrups.&nbsp; NONE of them can touch the excellence of this product.&lt;br /&gt;&lt;br /&gt;Thick, delicious.&nbsp; Perfect.&nbsp; 3 ingredients: Water, Maltitol, Natural Maple Flavor.&nbsp; PERIOD.&nbsp; No chemicals.&nbsp; No garbage.&lt;br /&gt;&lt;br/&gt; Unbelievably delicious&#8230;&lt;br /&gt;&lt;br /&gt; Can you tell I like it? 🙂



Preprocessing works like this:



1. Begin by removing the URLs and HTML tags



URLs like https://www.fakereviewsite.com in above example and HTML tags like &lt;br /&gt; &lt;br /&gt; don’t add any value to the review text. Think of this as doing “noise removal” for a piece of recorded audio.



Sample code:



<pre class="wp-block-code"><code>text_review = re.sub(r"http\S+", "", text_review)
text = BeautifulSoup(text_review, 'lxml').get_text()</code></pre>



Output:



Can&#8217;t do sugar.&nbsp; Have tried scores of SF Syrups.&nbsp; NONE of them can touch the excellence of this product.Thick, delicious.&nbsp; Perfect.&nbsp; 3 ingredients: Water, Maltitol, Natural Maple Flavor.&nbsp; PERIOD.&nbsp; No chemicals.&nbsp; No garbage. Unbelievably delicious&#8230; Can you tell I like it? 🙂



2. Expand the contractions (text standardization)



Contractions, the shortened/combined forms of commonly used words, are usually easier for humans to understand. We often think of them as making text and speech sound more “natural.” But for machines, using the full words allows them to handle sentiment analysis more easily. That’s why the next step involves expanding these contractions into complete words (e.g., “can’t” becomes “can not”). This is also called “text standardization.”



Sample code:



<pre class="wp-block-code"><code>text = re.sub(r"Can\'t", "can not", text)</code></pre>



Output:



can not do sugar.&nbsp; Have tried scores of SF Syrups.&nbsp; NONE of them can touch the excellence of this product.Thick, delicious.&nbsp; Perfect.&nbsp; 3 ingredients: Water, Maltitol, Natural Maple Flavor.&nbsp; PERIOD.&nbsp; No chemicals.&nbsp; No garbage. Unbelievably delicious&#8230; Can you tell I like it? 🙂



3. Remove numeric and alphanumeric words



Remove numbers and any word containing numbers, as in sentiment analysis numbers do not add any value.



Sample code:



<pre class="wp-block-code"><code>text = re.sub("\S*\d\S*", "", text).strip()</code></pre>



Output:



can not do sugar.&nbsp; Have tried scores of SF Syrups.&nbsp; NONE of them can touch the excellence of this product.Thick, delicious.&nbsp; Perfect. &nbsp; ingredients: Water, Maltitol, Natural Maple Flavor.&nbsp; PERIOD.&nbsp; No chemicals.&nbsp; No garbage. Unbelievably delicious&#8230; Can you tell I like it? 🙂



4. Remove punctuation and special characters



Remove all the punctuation and characters: pound signs (#), commas, periods, parentheses, etc. The text analysis we are about to perform will derive no value from these characters.



Sample code:



<pre class="wp-block-code"><code>text = re.sub('&#91;^A-Za-z0-9]+', ' ', text)</code></pre>



Output:



can not do sugar Have tried scores of SF Syrups NONE of them can touch the excellence of this product Thick delicious Perfect ingredients Water Maltitol Natural Maple Flavor PERIOD No chemicals No garbage Unbelievably delicious Can you tell I like it



5. Convert the text to lowercase



Next, we continue preprocessing the text by converting it all to lowercase. This is because the ML system interprets words with different cases as being different words. For example, “delicious” and “Delicious” appearing in the same text will be counted as two different words, as opposed to two instances of the same word!



6. Remove stop words



&#8220;Stop words&#8221; occur in every language and include common words like “we,” “they,” “can,” “not,” etc. These stop words do not add any value in NLP analysis, so we remove them. The Natural Language Toolkit (NLTK) list of English stop words can be found <a href="https://gist.github.com/sebleier/554280">here</a>.&nbsp;



However, be careful with stop words like “no” and “not,” as these words change the meaning of a sentence. For example, “tasty” and “not tasty” lead to two different sentiments.



So, we generally do not remove these specific stop words.



Sample code for text to lowercase and remove stop words:



<pre class="wp-block-code"><code>text = ' '.join(e.lower() for e in text.split() if e.lower() not in stopwords)</code></pre>



Output:



not sugar tried scores sf syrups none touch excellence product thick delicious perfect ingredients water maltitol natural maple flavor period no chemicals no garbage unbelievably delicious tell like



7. Lemmatization



Lemmatization means converting the word to its meaningful base form. For example, “tried” is turned into “try,” “cries” to “cry,” “cars” to “car,” etc. So, words like “tried,” “tries,” and “try” will all be considered by the system as multiple instances of the same word: “try.” Lemmatization can also be considered a part of normalization.



Sample code:



<pre class="wp-block-code"><code>lemmatizer = WordNetLemmatizer() 
tokenization = nltk.word_tokenize(text)
output = ' '.join(&#91;lemmatizer.lemmatize(w,pos="v") for w in tokenization])</code></pre>



Output:



not sugar try score sf syrup none touch excellence product thick delicious perfect ingredients water maltitol natural maple flavor period no chemical no garbage unbelievably delicious tell like



That does it for text preprocessing. Let’s move to the next best practice of NLP: data tokenization.



Deep data exploration: Advanced analytics and insights using Python and R



<a class="action-btn " href="https://www.sisense.com/whitepapers/deep-data-exploration-advanced-analytics-and-insights-using-python-and-r/" target="_blank" rel="noopener noreferrer">Read more</a>



<h2 class="wp-block-heading">Data tokenization</h2>



Tokenization is one of the most common best practices when working with text data. Tokenization means splitting the text into sentences or splitting the sentences into words. These split units are called tokens.



For example, the phrase “best practices in natural language processing” would be tokenized as: “best,” “practice,” “in,” “natural,” “language,” and “processing.”



In NLP, tokenization is important as the essence of a text can be easily interpreted by analysis of tokens present in it.



Sample code:



<pre class="wp-block-code"><code>tokenization = nltk.word_tokenize(output)</code></pre>



Output:



[&#8216;not&#8217;, &#8216;sugar&#8217;, &#8216;try&#8217;, &#8216;score&#8217;, &#8216;sf&#8217;, &#8216;syrups&#8217;, &#8216;none&#8217;, &#8216;touch&#8217;, &#8216;excellence&#8217;, &#8216;product&#8217;, &#8216;thick&#8217;, &#8216;delicious&#8217;, &#8216;perfect&#8217;, &#8216;ingredients&#8217;, &#8216;water&#8217;, &#8216;maltitol&#8217;, &#8216;natural&#8217;, &#8216;maple&#8217;, &#8216;flavor&#8217;, &#8216;period&#8217;, &#8216;no&#8217;, &#8216;chemicals&#8217;, &#8216;no&#8217;, &#8216;garbage&#8217;, &#8216;unbelievably&#8217;, &#8216;delicious&#8217;, &#8216;tell&#8217;, &#8216;like&#8217;]



<h2 class="wp-block-heading">Word embedding</h2>



Word embedding is an engineering technique in NLP where words or tokens are mapped to vectors of real numbers. In word embedding, each word converts into vectors with some dimensions, say “d.”



This process involves a learned representation for text where words or tokens that have the same meaning have a similar representation and can easily be understood by <a href="https://www.sisense.com/blog/deriving-value-from-data-lakes-with-artificial-intelligence/" target="_blank" rel="noreferrer noopener" aria-label="machine learning algorithms (opens in a new tab)">machine learning algorithms</a>.&nbsp;



Word embedding extracts meaning from text data as it also takes care of semantic relationships between genders, verb tense, and countries-capitals. 



There are various word embedding models available, such as word2vec by Google, Glove by Stanford, and fastText by Facebook.



For an example, let’s use Glove’s word embedding of 300 dimensions for the word “delicious.”



Sample code:



Embeddings_index[&#8220;delicious&#8221;]



Output:



array([ -0.27801&nbsp; , -0.14519&nbsp; ,&nbsp; 0.49453&nbsp; ,&nbsp; 0.12529&nbsp; , -0.057677 , 0.70151&nbsp; ,&nbsp; 0.28826&nbsp; , -0.20441&nbsp; ,&nbsp; 0.03009&nbsp; ,&nbsp; 1.3899 &nbsp; , -0.26564&nbsp; ,&nbsp; 0.43441&nbsp; , -0.47501&nbsp; , -0.13348&nbsp; ,&nbsp; 0.24737&nbsp; , -0.45528&nbsp; , -0.67027&nbsp; ,&nbsp; 1.1701 &nbsp; ,&nbsp; 0.040979 , -0.12553&nbsp; , -0.075785 , -0.27344&nbsp; , -0.049158 , -0.42694&nbsp; , -0.041506 , -0.11606&nbsp; ,&nbsp; 0.3838 &nbsp; , -0.1245 &nbsp; ,&nbsp; 0.018793 , -0.78534&nbsp; , 0.055234 ,&nbsp; 0.007924 ,&nbsp; 0.055043 , -0.20598&nbsp; , -0.06414&nbsp; , -0.10594&nbsp; , -0.06305&nbsp; , -0.027444 , -0.26364&nbsp; ,&nbsp; 0.64279&nbsp; , -0.29369&nbsp; ,&nbsp; 0.11132&nbsp; ,&nbsp; 0.28754&nbsp; , -0.27219&nbsp; ,&nbsp; 0.48337&nbsp; , 0.93093&nbsp; , -0.23844&nbsp; ,&nbsp; 0.61936&nbsp; ,&nbsp; 0.12593&nbsp; , -0.24751&nbsp; , -0.08677&nbsp; ,&nbsp; 0.19172&nbsp; , -0.36446&nbsp; , -0.071028 ,&nbsp; 0.64807&nbsp; , 0.12868&nbsp; , -0.046247 ,&nbsp; 0.42061&nbsp; , -0.12793&nbsp; ,&nbsp; 0.19642&nbsp; , &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0.68146&nbsp; , -0.55865&nbsp; , -0.27874&nbsp; ,&nbsp; 0.039101 , -0.17919&nbsp; , -0.59897&nbsp; ,&nbsp; 0.20486&nbsp; ,&nbsp; 0.15241&nbsp; ,&nbsp; 0.34993&nbsp; ,&nbsp; 0.47898&nbsp; , 0.36544&nbsp; ,&nbsp; 0.57892&nbsp; ,&nbsp; 0.24779&nbsp; , -0.35317&nbsp; ,&nbsp; 0.2616 &nbsp; , -0.22896&nbsp; , -0.22391&nbsp; ,&nbsp; 0.16569&nbsp; , -0.61168&nbsp; , -0.18378&nbsp; , -0.023205 , -0.18056&nbsp; ,&nbsp; 0.054312 , -0.1776 &nbsp; , -0.098411 , -0.6113 &nbsp; ,&nbsp; 0.38856&nbsp; ,&nbsp; 0.88379&nbsp; , -0.29055&nbsp; , -0.12958&nbsp; , 0.015754 ,&nbsp; 0.23812&nbsp; ,&nbsp; 0.10429&nbsp; ,&nbsp; 0.41016&nbsp; ,&nbsp; 0.23708&nbsp; , 1.0123 &nbsp; , -0.86614&nbsp; , -0.16838&nbsp; ,&nbsp; 0.066406 , -0.0050272, -0.22711&nbsp; , -0.28863&nbsp; ,&nbsp; 0.36877&nbsp; , -0.25895&nbsp; , -0.22054&nbsp; , -0.31888&nbsp; ,&nbsp; 0.58853&nbsp; ,&nbsp; 0.21332&nbsp; ,&nbsp; 0.55837&nbsp; , -0.23193&nbsp; , -0.21208&nbsp; , -0.56664&nbsp; , -0.66216&nbsp; ,&nbsp; 0.22095&nbsp; ,&nbsp; 0.12373&nbsp; , -0.48547&nbsp; , -0.44839&nbsp; ,&nbsp; 0.0091947, -0.27908&nbsp; ,&nbsp; 0.014443 , 0.21652&nbsp; ,&nbsp; 0.18283&nbsp; , -0.35423&nbsp; , -0.4034 &nbsp; ,&nbsp; 0.27554&nbsp; , -0.52523&nbsp; ,&nbsp; 0.436&nbsp; &nbsp; , -0.60157&nbsp; ,&nbsp; 0.18374&nbsp; ,&nbsp; 0.11548&nbsp; , -0.17291&nbsp; , -0.89038&nbsp; ,&nbsp; 0.38101&nbsp; ,&nbsp; 0.32373&nbsp; , -0.25688&nbsp; , 0.19965&nbsp; , -0.11587&nbsp; ,&nbsp; 0.025945 , -0.041164 , -0.31512&nbsp; , -2.5693 &nbsp; ,&nbsp; 0.35069&nbsp; ,&nbsp; 0.57936&nbsp; ,&nbsp; 0.24183&nbsp; , -0.025946 , -1.205&nbsp; &nbsp; , -0.061434 ,&nbsp; 0.057562 ,&nbsp; 0.55008&nbsp; , -0.016724 , 0.32504&nbsp; ,&nbsp; 0.11345&nbsp; ,&nbsp; 0.40961&nbsp; ,&nbsp; 0.16263&nbsp; ,&nbsp; 0.19778&nbsp; , -0.017293 , -0.12313&nbsp; , -0.1714 &nbsp; ,&nbsp; 0.80623&nbsp; , -0.039624 , -0.39989&nbsp; ,&nbsp; 0.52971&nbsp; ,&nbsp; 0.42122&nbsp; ,&nbsp; 0.077219 , -0.005385 , -0.63461&nbsp; ,&nbsp; 0.25213&nbsp; , -0.37005&nbsp; ,&nbsp; 0.20136&nbsp; , -0.052806 , 0.59213&nbsp; , -0.12338&nbsp; ,&nbsp; 0.22055&nbsp; , -0.195&nbsp; &nbsp; , -0.66998&nbsp; , -0.18867&nbsp; ,&nbsp; 0.022199 , -0.82456&nbsp; , -0.080926 , -0.41921&nbsp; , 0.034355 , -0.69545&nbsp; ,&nbsp; 0.24931&nbsp; , -0.10916&nbsp; ,&nbsp; 0.12605&nbsp; , -0.75361&nbsp; , -0.033696 , -0.2305 &nbsp; , -0.0046053, -0.31902&nbsp; , -0.31114&nbsp; , -0.036903 , -0.022895 , -0.13569&nbsp; , -0.27607&nbsp; , -0.1031 &nbsp; ,&nbsp; 0.30701&nbsp; ,&nbsp; 0.34186&nbsp; ,&nbsp; 0.45891&nbsp; , -0.13587&nbsp; , -0.45223&nbsp; , -0.090191 , -0.099395 , -0.074891 ,&nbsp; 0.19245&nbsp; , 0.29045&nbsp; , -0.39008&nbsp; ,&nbsp; 0.73567&nbsp; ,&nbsp; 0.62942&nbsp; , -0.2174 &nbsp; , -0.51588&nbsp; , -0.19546&nbsp; ,&nbsp; 0.081308 , -0.030894 , -0.34068&nbsp; , 0.28449&nbsp; ,&nbsp; 0.40505&nbsp; , -0.33851&nbsp; ,&nbsp; 0.24456&nbsp; ,&nbsp; 0.093235 , -0.53475&nbsp; ,&nbsp; 0.34995&nbsp; ,&nbsp; 0.1656 &nbsp; ,&nbsp; 0.80673&nbsp; ,&nbsp; 0.48231&nbsp; , -0.39488&nbsp; , -0.20581&nbsp; ,&nbsp; 0.063178 ,&nbsp; 0.065316 ,&nbsp; 0.17051&nbsp; , -0.16726&nbsp; , -0.28956&nbsp; , -0.50795&nbsp; , -0.39699&nbsp; ,&nbsp; 0.19386&nbsp; ,&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -0.16445&nbsp; , -0.4318 &nbsp; , -0.47626&nbsp; , -0.20233&nbsp; ,&nbsp; 0.11089&nbsp; , 0.13755&nbsp; ,&nbsp; 0.026714 , -0.93893&nbsp; , -0.20077&nbsp; ,&nbsp; 0.20623&nbsp; , 0.7216 &nbsp; , -0.58006&nbsp; , -0.38965&nbsp; , -0.2282 &nbsp; ,&nbsp; 0.17188&nbsp; , -0.3815 &nbsp; ,&nbsp; 0.04917&nbsp; , -0.32791&nbsp; ,&nbsp; 0.10739&nbsp; ,&nbsp; 0.023031 , 0.67157&nbsp; ,&nbsp; 0.32911&nbsp; ,&nbsp; 0.28143&nbsp; ,&nbsp; 0.036222 ,&nbsp; 0.1453 &nbsp; , -0.12512&nbsp; , -0.27149&nbsp; ,&nbsp; 0.04054&nbsp; , -0.020042 , -0.056311 , 0.34275&nbsp; , -0.62091&nbsp; , -0.0058532, -0.49363&nbsp; ,&nbsp; 0.072698 , -0.81502&nbsp; , -0.026662 , -0.23517&nbsp; , -0.34235&nbsp; , -0.54425&nbsp; , 0.45515&nbsp; ,&nbsp; 0.085665 ,&nbsp; 0.070533 ,&nbsp; 0.36966&nbsp; ,&nbsp; 0.95099&nbsp; , 0.47395&nbsp; , -0.1195 &nbsp; ,&nbsp; 0.12501&nbsp; , -0.50397&nbsp; ,&nbsp; 0.10813&nbsp; , 0.53519&nbsp; ,&nbsp; 0.58557&nbsp; ,&nbsp; 0.56703&nbsp; , -0.17101&nbsp; ,&nbsp; 0.48838&nbsp; , 0.46119&nbsp; ,&nbsp; 0.4737 &nbsp; , -0.14692&nbsp; ,&nbsp; 0.0055303, -0.37672&nbsp; , -0.090149 ,&nbsp; 0.52314&nbsp; , -0.97767&nbsp; ,&nbsp; 0.18443&nbsp; ,&nbsp; 0.25023&nbsp; ], dtype=float32)



The vectors for tokens from word embedding are used for the next ML step (not in this exercise), modeling. Modeling is where we teach a program to learn from training data using ML algorithms.



<h2 class="wp-block-heading">Proper preparation, accurate execution</h2>



Now we’ve got a properly prepared text dataset: Unnecessary filler words have been removed, all text is lowercase and in its simplest form, and words have been assigned values that our ML algorithms can work with. These best practices will help you get the most from your text analysis, pulling sentiments out of human words without having to read each one yourself. And that’s just the start of what ML can do for you. Happy coding!



Deep data exploration: Advanced analytics and insights using Python and R



<a class="action-btn " href="https://www.sisense.com/whitepapers/deep-data-exploration-advanced-analytics-and-insights-using-python-and-r/" target="_blank" rel="noopener noreferrer">Read more</a>



Nidhi Bansal&nbsp;is Data Scientist, Machine Learning/Artificial Intelligence enthusiast, and writer who loves to experiment with data and write about it. She has over a decade of experience in software development in various programming languages and holds a B.Tech and M.E in Electronics and Communications Engineering.

Natural Language Processing Best Practices in Python and R

LinkedIn

Twitter

GitHub

curve-image-unique-image-unique

curve

3-dark-2-image-unique-image-unique

3 DARK 2

Get the latest in analytics right in your inbox.

Article