Accessing <li> element with no class id using Beautiful soup
<p>I am trying to scrape the companies in the li in the ul table under final result. The source code looks like this</p>

<p>import string
import re
import urllib2
import datetime
import bs4
from bs4 import BeautifulSoup</p>

<p>class AJSpider(object):</p>

<pre><code>def __init__(self):
    print ("initisizing")
    self.date = str(datetime.date.today())
    self.cur_url = "https://youinvest.moneyam.com/modules/forward-diary/?date={date}&amp;period=month"
    self.datas = []
    print ("initisization done")


def get_page(self,cur_date):
    url = self.cur_url
    try:
        my_page = urllib2.urlopen(url.format(date = cur_date)).read().decode("utf-8")
        my_soup = BeautifulSoup(my_page, "html.parser")

    except:
        print ('Failed')
    return my_soup

def get_final(self, soup_page):
    temp_data = []
    final_result_section = soup_page.find("h3", text="Final Result")
    print final_result_section

def start_spider(self):
    my_page = self.get_page(self.date)
    self.get_final(my_page)

def main():

    my_spider = AJSpider()
    my_spider.start_spider()

if __name__ == '__main__':
    main()
</code></pre>

<p>I found a similar quesiton in stackoverflow 
<a href="http://stackoverflow.com/questions/17683475/beautiful-soup-accessing-li-elements-from-ul-with-no-id">Beautiful Soup: Accessing &lt;li&gt; elements from &lt;ul&gt; with no id</a> ,  but this one here does have a class id, which makes things a lot easier.</p>

<p>In my scenario, how may I extract the li element from the ul table please? the only identifier here is really the content of the h3 tag, which is Final Result, however it is not a id so I have no idea how to make use of it.</p>

<p>Many thanks!</p>

<p>Find the <code>h3</code> element <em>by text</em> and get the <a href="https://www.crummy.com/software/BeautifulSoup/bs4/doc/#find-next-siblings-and-find-next-sibling" rel="nofollow">following <code>ul</code> list</a>:</p>

<pre><code>ul = soup.find("h3", text="Final Result").find_next_sibling("ul")
for li in ul.find_all("li"):
    print(li.span.get_text(), li.a.get_text())
</code></pre>

<hr>

<p>Note that in the recent versions of BeautifulSoup, <code>text</code> argument was renamed to <a href="https://www.crummy.com/software/BeautifulSoup/bs4/doc/#the-string-argument" rel="nofollow"><code>string</code></a>, but they both work because of the backwards compatibility.</p>

python html beautifulsoup html-lists