parse html tables with lxml
<p>I have been trying to parse the table contents from <a href="https://nseindia.com/products/content/derivatives/equities/fo_underlying_home.htm" rel="nofollow">here</a>
i have tried a couple of alternatives, like </p>

<pre><code>xpath('//table//tr/td//text()') 
xpath('//div[@id="replacetext"]/table/tbody//tr/td/a//text()')
</code></pre>

<p>here is my last code:</p>

<pre><code>import requests, lxml.html
url ='https://nseindia.com/products/content/derivatives/equities/fo_underlying_home.htm'
url = requests.get(url)
html = lxml.html.fromstring(url.content)
packages = html.xpath('//div[@id="replacetext"]/table/tbody//tr/td/a//text()') # get the text inside all "&lt;tr&gt;&lt;td&gt;&lt;a ...&gt;text&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;"
</code></pre>

<p>however none of the alternatives seems to be working. In the past, i have scraped data with similar code (although not from this url!). Any guidance will be really helpful.</p>

<p>In the HTML page, there is a namespace:</p>

<pre><code>&lt;html xmlns="http://www.w3.org/1999/xhtml"&gt;
</code></pre>

<p>So, you need to specify it:</p>

<pre><code>NSMAP = {'html' : "http://www.w3.org/1999/xhtml"}

path = '//html:div[@id="replacetext"]/html:table/html:tbody//html:tr/html:td/html:a//text()'
packages = html.xpath(path, namespaces=NSMAP)
</code></pre>

<p>See <a href="http://lxml.de/xpathxslt.html#namespaces-and-prefixes" rel="nofollow">http://lxml.de/xpathxslt.html#namespaces-and-prefixes</a></p>

<h2>Interpreting Ajax call</h2>

<pre><code>import requests
from lxml import html

base_url = 'https://nseindia.com'

# sumulate the JavaScript
url = base_url + "/products/content/derivatives/equities/fo_underlyinglist.htm"
url = requests.get(url)
content = url.content
# -&gt; &lt;table&gt;
#    &lt;tr&gt;&lt;th&gt;S. No.&lt;/td&gt;
#    &lt;th&gt;Underlying&lt;/td&gt;
#    &lt;th&gt;Symbol&lt;/th&gt;&lt;/tr&gt;
#    &lt;tr&gt;
#    &lt;td style='text-align: right;' &gt;1&lt;/td&gt;
#    &lt;td class="normalText" &gt;&lt;a href=fo_INDIAVIX.htm&gt;INDIA VIX&lt;/a&gt;&lt;/td&gt;
#    &lt;td class="normalText" &gt;&lt;a href="/products/dynaContent/derivatives/equities/fomwatchsymbol.jsp?key=INDIAVIX"&gt;INDIAVIX&lt;/a&gt;&lt;/td&gt;
#    &lt;/tr&gt;
#    ...

html = html.fromstring(content)
packages = html.xpath('//td/a//text()')
# -&gt; ['INDIA VIX',
#     'INDIAVIX',
#     'Nifty 50',
#     'NIFTY',
#     'Nifty IT',
#     'NIFTYIT',
#     'Nifty Bank',
#     'BANKNIFTY',
#     'Nifty Midcap 50',
</code></pre>

python parsing lxml