How to get these information in context of given below using beautiful soup python
<p>Going to  <a href="https://www.couponraja.in" rel="nofollow">https://www.couponraja.in</a></p>

<p>I need to data scrape the whole website with the code:
As I wrote my code like this:</p>

<pre><code>wiki = 'https://www.couponraja.in'
page = urllib2.urlopen(wiki)
soup = BeautifulSoup(page, 'html.parser')

titles = soup.find_all('p', attrs={'class':'ofr-sml-descp-nw'})
for p in titles:
    print 'title: ',p.text



divs = soup.select('div.deal-sub-nw')
for x in divs:
    print 'Offer-Title:', x.select('p.ofr-sml-descp-nw')[0].text
    print 'Offer-Image URL:', x.select('img')[0].attrs['src']

div1 = soup.select('div.m-sml-logo-nwcell')
for x in div1:
    print 'Offer-Merchant:',x.select('a')[0].attrs['alt']





#for a in soup.find_all('a', href=True):
    #print "Found the URL:", a['href']

#print(soup.prettify())

#text = soup.find('p',{'class':'ofr-sml-descp-nw'}) --&gt; prints single statement
</code></pre>

<ol>
<li><p>OFFER TITLE </p></li>
<li><p>OFFER DESCRIPTION</p></li>
<li><p>OFFER TERMS AND CONDITIONS  --> as i want to print pop up, but it only get me with Terms and Condition,i.e. i can't print the pop up frame which is with it.</p></li>
<li><p>OFFER START DATE</p></li>
<li><p>OFFER END DATE</p></li>
<li><p>OFFER URL</p></li>
<li><p>OFFER MERCHANT</p></li>
<li><p>OFFER IMAGE URL (ANY ONE) </p></li>
<li><p>OFFER AREA (EXAMPLE NAME OF SUBURB)  - APPLICABLE FOR RETAIL OFFERS </p></li>
<li><p>OFFER CITY - APPLICABLE FOR RETAIL OFFERS</p></li>
<li><p>OFFER PINCODE  - APPLICABLE FOR RETAIL OFFERS</p></li>
<li><p>OFFER ADDRESS - APPLICABLE FOR RETAIL OFFERS</p></li>
<li><p>OFFER CATEGORY</p></li>
<li><p>OFFER BANK (ICICI Card, PAYTM etc) </p></li>
</ol>

<p>Please Help me to get these, as I am a newbie in it and want to know how it works.
Thanks For the Help</p>

<p>I don't really see anything wrong with your code. Perhaps the approach can be modified.  If your going to use a for loop for all the items you are parsing it going to take code. If things on that page are being generated with javascript, then you can't use the raw html you get from a get request, you'll probably have to use something like selenium.  Following your approach I'd continue with something like this. Lastly, look into using regexes, you know what you're looking for internally.. So instead of writing for loop after for loop, you can probably write a regexes to get items in a few passes. I'm thinking O^2 runtime, then agin there still might be a better approach.  That's the beauty of programming, if you get stuck look at it differently.</p>

<pre><code>import json
import requests
import sys

from bs4 import BeautifulSoup

url = 'https://www.couponraja.in'
res = requests.get(url)
if res.status_code != 200:
  print "Page Could not be accessed"
  sys.exit(1)

data = {}
titles = []
desciptions = []
# add more list

soup = BeautifulSoup(res.content, 'html.parser')
t = soup.find_all('p', attrs={'class': 'ofr-sml-descp-nw'})
for p in t:
  try:
    titles.append(p.contents)
  except AttributeError:
    pass



# i'm assumming the two list are the same length you'll have to add your own checks
data = []
for i in xrange(len(titles)):
  temp = {}
  try:
    temp['title'] = titles[i]
    # add more fields
  except IndexError:
    pass

  data.append(temp)

print json.dumps(data, indent=2)
</code></pre>

<p>You just need to keep at it.  No one on here is going to parse that data for you... and your not asking about a specific issue.</p>

<p>Given you're comments below, this is how to extract the content from the 'rvwssec' class, which contains the word Terms and Conditions.</p>

<pre><code>terms = soup.find_all('div', attrs={'class': 'rvwssec'})
for term in terms:
  print term.findNext('a').contents[0]
</code></pre>

<p>Now you'll have to convert it to a json yourself, you can store the data in a list like I showed above, convert those list into a single dictionary.  Python Dictionaries and JSON are very similar. But to ensure that everything is proper json, you can call json.dumps(my_dictionary) to make the full conversion.</p>

python html beautifulsoup