List of list of words by Python:
<p>Having a long list  of comments (50 by saying) such as this one:   </p>

<blockquote>
  <p>"this was the biggest disappointment of our trip. the restaurant had
  received some very good reviews, so our expectations were high. the
  service was slow even though the restaurant was not very full. I had
  the house salad which could have come out of any sizzler in the us.
  the keshi yena, although tasty reminded me of barbequed pulled
  chicken. this restaurant is very overrated".</p>
</blockquote>

<p>I want to create a list of list of words retaining sentence tokenization using python.  </p>

<p>After removing stopwords I want a result for all 50 comments in which sentence tokens are retained and word tokens are retained into each tokenized sentence. At the end I hope result being similar to:  </p>

<pre><code>list(c("disappointment", "trip"), 
     c("restaurant", "received", "good", "reviews", "expectations", "high"), 
     c("service", "slow", "even", "though", "restaurant", "full"),
     c("house", "salad", "come", "us"), 
     c("although", "tasty", "reminded", "pulled"), 
     "restaurant")  
</code></pre>

<p>How could I do that in python? Is R a good option in this case? I really will appreciate your help.</p>

<p>This is one way to do it. You may need to initialize the <code>stop_words</code> as suits your application. I have assumed <code>stop_words</code> is in lowercase: hence, using <code>lower()</code> on the original sentences for comparison. <code>sentences.lower().split('.')</code> gives sentences. <code>s.split()</code> gives list of words in each sentence.</p>

<pre><code>stokens = [list(filter(lambda x: x not in stop_words, s.split())) for s in sentences.lower().split('.')]
</code></pre>

<p>You may wonder why we use <code>filter</code> and <code>lambda</code>. An alternative is this but this will give a flat list and hence is not suitable:</p>

<pre><code>stokens = [word for s in sentences.lower().split('.') for word in s.split() if word not in stop_words]
</code></pre>

<p><code>filter</code> is a functional programming construct. It helps us to process an entire list, in this case, via an anonymous function using the <code>lambda</code> syntax.</p>

python word-list