Wikipedia table scraping using python
<p>I am trying to scrape tables from <a href="https://en.wikipedia.org/wiki/List_of_countries_and_dependencies_by_population" rel="nofollow">wikipedia</a>. I wrote a table scrapper using tutorials available in the web which downloads table and save as pandas data frame.</p>

<p>This is the code</p>

<pre><code>from bs4 import BeautifulSoup
import pandas as pd
import urllib2

headers = { 'User-Agent' : 'Mozilla/5.0' }
req = urllib2.Request('https://en.wikipedia.org/wiki/List_of_countries_and_dependencies_by_population', None, headers)
html = urllib2.urlopen(req).read()
soup = BeautifulSoup(html, 'lxml') # Parse the HTML as a string
print soup

# Create an object of the first object 
table = soup.find("table", {"class":"wikitable sortable jquery-tablesorter"})
print table

rank=[]
country=[]
pop=[]
date=[]
per=[]
source=[]

for row in table.find_all('tr')[1:]:
    col=row.find_all('td')
    col1=col[0].string.strip()
    rank.append(col1)
    col2=col[1].string.strip()
    country.append(col2)
    col3=col[2].string.strip()
    pop.append(col2)
    col4=col[3].string.strip()
    date.append(col4)
    col5=col[4].string.strip()
    per.append(col5)
    col6=col[5].string.strip()
    source.append(col6)

columns={'Rank':rank,'Country':country,'Population':pop,'Date':date,'Percentage':per,'Source':source}
# Create a dataframe from the columns variable
df = pd.DataFrame(columns)
df
</code></pre>

<p>But it is not downloading the table. The problem is in this section</p>

<pre><code>table = soup.find("table", {"class":"wikitable sortable jquery-tablesorter"})
print table
</code></pre>

<p>where output is <code>None</code></p>

<p>As far as I can see, there is no such element on that page. The main table has <code>"class":"wikitable sortable"</code> but not the <code>jquery-tablesorter</code>.</p>

<p>Make sure you know what element you are trying to select and check if your program sees the same elements you see, then make your selector.</p>

python web-scraping beautifulsoup