Crawl a wikipedia page in python and store it in a .csv file
<p>This is my python script code</p>

<pre><code>from bs4 import BeautifulSoup 
import requests
url='https://en.wikipedia.org/wiki/List_of_Pok%C3%A9mon'
source_code=requests.get(url)
print(url)
plain_text=source_code.text
soup=BeautifulSoup(plain_text,"html.parser")
print("hello")
f = open('pokemons.csv', 'w')
for table2 in soup.findAll('table'):
        print("yash")
        for i in table2.findAll('tbody'):
            print("here")
            for link in i.findAll('tr'):                
                for x in link.findAll('td'):
                        for y in x.findAll('a'):
                                z=y.get('href')
                                print(z)
                                #f.write(link)
f.close()
</code></pre>

<p>All i want to do is crawl all the pokemons name from this wiki link
<a href="https://en.wikipedia.org/wiki/List_of_Pok%C3%A9mon" rel="nofollow">https://en.wikipedia.org/wiki/List_of_Pok%C3%A9mon</a>
but problem here is i am unable to get into the specified table i. e the table in which the all the pokemons names are stored but im my above code i m travelling through alll the table and trying to acces the "tbody" tag so that i will access "tr" tags in it but it not happening the same way !! tell me my mistake.</p>

<p>The <em>tbody</em> is added by the browser so not in the actual source returned by requests so your code could never find anything using it.</p>

<p>All you need to do is get every <em>anchor</em> with a <em>title</em> attribute in each row:</p>

<pre><code>with open('pokemons.csv', 'w') as f:
    table = soup.select_one("table.wikitable.sortable")
    for a in table.select("tr td a[title]"):
         f.write(a.text.encode("utf-8") + "\n")
</code></pre>

<p>That gives you 761 names, one per line.</p>

<p>If you were to use find_all and find, it would be like: </p>

<pre><code> # get all orws
 for tr in table.find_all("tr"):
    # see if there is an anchor inside with a title attribute
    a = tr.find("a", title=True)
    # if there is write the text
    if a:
        f.write(tr.find("a", title=True).text.encode("utf-8") + "\n")
</code></pre>

python beautifulsoup web-crawler