Unable to scrape this movie website using BeautifulSoup
<p>I am trying to scrap a movie website here: <a href="http://www.21cineplex.com/nowplaying" rel="nofollow">http://www.21cineplex.com/nowplaying</a></p>

<p>I have uploaded the screenshot with the HTML body as the image in this questions.<a href="http://i.stack.imgur.com/OncE1.png" rel="nofollow">link to screenshot here</a> I am having difficulty trying to grab the movie title and the description which is part of the <code>&lt;P&gt;</code> tag. For some strange reason, the description is not part of requests object. Also when I tried to use soup to find the ul and class name it cannot be found. Anyone know why? I am using python 3. This is my code so far:</p>

<pre><code>    r = requests.get('http://www.21cineplex.com/nowplaying')
    r.text (no description here)
    soup = bs4.BeautifulSoup(r.text)
    soup.find('ul', class_='w462') # why is this empty?
</code></pre>

<p>This server is checking <code>Referer</code> header. If there is no <code>Referer</code> it sends main page. But it doesn't check text in this header so it can be even empty string.</p>

<pre><code>import requests
import bs4

headers = {
    #'Referer': any url (or even random text, or empty string)

    #'Referer': 'http://google.com',
    #'Referer': 'http://www.21cineplex.com',
    #'Referer': 'hello world!',
    'Referer': '',
}

s = requests.get('http://www.21cineplex.com/nowplaying', headers=headers)
soup = bs4.BeautifulSoup(s.text)

for x in soup.find_all('ul', class_='w462'):
    print(x.text)

for x in soup.select('ul.w462'):
    print(x.text)

for x in soup.select('ul.w462'):
    print(x.select('a')[0].text)
    print(x.select('p')[0].text)
</code></pre>

python html web-scraping beautifulsoup python-requests