Why is BeautifulSoup not extracting all of HTML from a webpage?
<p>I am trying to extract text from this website: <a href="https://www.searchgurbani.com/guru_granth_sahib/ang_by_ang" rel="nofollow">searchgurbani</a>. This website has some old scripture translated in English and Punjabi (an Indian Language) line-by-line. It makes a very good parallel corpus. I have successfully extracted all the English translations in a separate text file. But when I go for Punjabi, It returns nothing.</p>

<p>This is the Inspect element screenshot: (Highlighted text is the translated Punjabi language)</p>

<p><a href="http://i.stack.imgur.com/Jzai7.png" rel="nofollow">Screenshot 1</a></p>

<p>In Screenshot 1, highlighted text which belongs to <em>class=lang_16</em> is not listed in the soup object <em>beautiful</em> which should contain all of the HTML. Here is the Python code:</p>

<pre><code>outputFilePunjabi = open("1.txt","w",newline="",encoding="utf-16")
r=urlopen("")
beautiful = BeautifulSoup(r.read().decode('utf-8'),"html5lib")
#beautiful = BeautifulSoup(r.read().decode('utf-8'),"lxml")
punjabi_text = beautiful.find_all(class_="lang_16")
for i in punjabi_text:
  outputFilePunjabi.write(i.get_text())
  outputFilePunjabi.write('\n')
</code></pre>

<p>If I run the same code with <em>class_=lang_4</em> it does the work.</p>

<p>Please do the following to see lang_16 in inspect element:</p>

<p><em>Please do the following on that web page: Go to preferences --> Tick "translation of Sri Guru Granth Sahib ji (by S. Manmohan Singh) - Punjabi" under Additional Translations available on Guru Granth Shahib: --> scroll down - submit changes -> reopen page</em></p>

<p>Please guide me where I am going wrong.</p>

<p>(python version = 3.5)</p>

<p><em>PS: I have very less experience in web scrapping.</em></p>

<p>Remember you've suggested to do the following:</p>

<blockquote>
  <p>Please do the following on that web page: Go to preferences -> Tick
  "ranslation of Sri Guru Granth Sahib ji (by S. Manmohan Singh) -
  Punjabi" under Additional Translations available on Guru Granth
  Shahib: -> scroll down - submit changes</p>
</blockquote>

<p>Now, this is also required when you download the page in Python. In other words, use <a href="http://docs.python-requests.org/en/master/" rel="nofollow"><code>requests</code></a> and <strong>set the <code>lang_16="yes"</code> cookie</strong> to enable the Punjabi translation: </p>

<pre><code>import requests
from bs4 import BeautifulSoup


with requests.Session() as session:
    response = session.get("https://www.searchgurbani.com/guru_granth_sahib/ang_by_ang", cookies={
        "lang_16": "yes"
    })
    soup = BeautifulSoup(response.content, "html5lib")
    for item in soup.select(".lang_16"):
        print(item.get_text())
</code></pre>

<p>Prints:</p>

<pre><code>à¨µà¨¾à¨¹à¨¿à¨à©à¨°à© à¨à©à¨µà¨² à¨à¨ à¨¹à©à¥¤ à¨¸à©±à¨à¨¾ à¨¹à© à¨à¨¸ à¨¦à¨¾ à¨¨à¨¾à¨®, à¨°à¨à¨¨à¨¹à¨¾à¨° à¨à¨¸ à¨¦à© à¨µà¨¿à¨à¨à¨¤à© à¨à¨¤à© à¨à¨®à¨° à¨à¨¸ à¨¦à¨¾ à¨¸à¨°à©à¨ªà¥¤ à¨à¨¹ à¨¨à¨¿à¨¡à¨°, à¨à©à¨¨à¨¾-à¨°à¨¹à¨¿à¨¤, à¨à¨à¨¨à¨®à¨¾ à¨¤à© à¨¸à¨µà©-à¨ªà©à¨°à¨à¨¾à¨¶à¨µà¨¾à¨¨ à¨¹à©à¥¤ à¨à©à¨°à¨¾à¨ à¨¦à© à¨¦à¨¯à¨¾ à¨¦à©à¨à¨°à¨¾ à¨à¨¹ à¨ªà¨°à¨¾à¨ªà¨¤ à¨¹à©à©°à¨¦à¨¾ à¨¹à©à¥¤
à¨à¨¸ à¨¦à¨¾ à¨¸à¨¿à¨®à¨°à¨¨ à¨à¨°à¥¤
à¨ªà¨°à¨¾à¨°à©°à¨­ à¨µà¨¿à©±à¨ à¨¸à©±à¨à¨¾, à¨¯à©à¨à¨¾à¨ à¨¦à© à¨¶à©à¨°à© à¨µà¨¿à©±à¨ à¨¸à©±à¨à¨¾,
à¨à¨¤à© à¨¸à©±à¨à¨¾ à¨à¨¹ à¨¹à©à¨£ à¨­à© à¨¹à©, à¨¹à© à¨¨à¨¾à¨¨à¨! à¨¨à¨¿à¨¸à¨à¨¿à¨¤ à¨¹à©, à¨à¨¹ à¨¸à©±à¨à¨¾ à¨¹à©à¨µà©à¨à¨¾à¥¤
...
à¨à¨ à¨à¨ à¨à¨¾à¨à¨¨ à¨à¨°à¨¦à© à¨¹à¨¨ à¨à¨¿ à¨µà¨¾à¨¹à¨¿à¨à©à¨°à© à¨ªà©à¨°à¨¾à¨£ à¨²à© à¨²à©à¨à¨¦à¨¾ à¨¹à© à¨¤à© à¨®à©à© à¨µà¨¾à¨ªà¨¸ à¨¦à© à¨¦à¨¿à©°à¨¦à¨¾ à¨¹à©à¥¤
à¨à¨ à¨à¨¾à¨à¨¨ à¨à¨°à¨¦à© à¨¹à¨¨ à¨à¨¿ à¨¹à¨°à© à¨¦à©à¨°à©à¨¡à© à¨®à¨²à©à¨® à¨¹à©à©°à¨¦à¨¾ à¨à¨¤à© à¨¸à©à©±à¨à¨¦à¨¾ à¨¹à©à¥¤
</code></pre>

python html python-3.x web-scraping beautifulsoup