Python - BeautifulSoup scrape non-standard web table
<p>I am attempting to scrape data from several webpages in order to create a CSV of the data. The data is just nutritional information of products. I have generated code to access the website, but I can't quite get the code to iterate out properly. The problem is, the website uses DIV tags for the product name, and inside of the DIV either  or , it varies between the pages. When I attempt to iterate it out, the product names all show at once, in a list with tags, and then I get the contents of the column I requested, without tags. I am trying to figure out what I am doing wrong. </p>

<p>Source code example:</p>

<pre><code>&lt;div&gt;&lt;strong&gt;Product 1 Name&lt;/strong&gt;&lt;/div&gt;

&lt;table&gt;
    &lt;tbody&gt;
        &lt;tr&gt;
            &lt;td&gt;Serving Size&lt;/td&gt;
            &lt;td&gt;8 (fl. Oz.)&lt;/td&gt;
        &lt;/tr&gt;
        &lt;tr&gt;
            &lt;td&gt;Calories&lt;/td&gt;
            &lt;td&gt;122 Calories&lt;/td&gt;
        &lt;/tr&gt;
        &lt;tr&gt;
            &lt;td&gt;Fat&lt;/td&gt;
            &lt;td&gt;0 (g)&lt;/td&gt;
        &lt;/tr&gt;
        &lt;tr&gt;
            &lt;td&gt;Sodium&lt;/td&gt;
            &lt;td&gt;0.2 (mg)&lt;/td&gt;
        &lt;/tr&gt;
        &lt;tr&gt;
            &lt;td&gt;Carbs&lt;/td&gt;
            &lt;td&gt;8.8 (mg)&lt;/td&gt;
        &lt;/tr&gt;
        &lt;tr&gt;
            &lt;td&gt;Dietary Fiber&lt;/td&gt;
            &lt;td&gt;0 (g)&lt;/td&gt;
        &lt;/tr&gt;
        &lt;tr&gt;
            &lt;td&gt;Sugar&lt;/td&gt;
            &lt;td&gt;8.8 (g)&lt;br /&gt;
            &amp;nbsp;&lt;/td&gt;
        &lt;/tr&gt;
    &lt;/tbody&gt;
&lt;/table&gt;
&amp;nbsp;

&lt;div&gt;&lt;strong&gt;Product 2 Name&lt;/strong&gt;&lt;/div&gt;

&lt;table&gt;
    &lt;tbody&gt;
        &lt;tr&gt;
            &lt;td&gt;Serving Size&lt;/td&gt;
            &lt;td&gt;8 (fl. Oz.)&lt;/td&gt;
        &lt;/tr&gt;
        &lt;tr&gt;
            &lt;td&gt;Calories&lt;/td&gt;
            &lt;td&gt;134 Calories&lt;/td&gt;
        &lt;/tr&gt;
        &lt;tr&gt;
            &lt;td&gt;Fat&lt;/td&gt;
            &lt;td&gt;0 (g)&lt;/td&gt;
        &lt;/tr&gt;
        &lt;tr&gt;
            &lt;td&gt;Sodium&lt;/td&gt;
            &lt;td&gt;0.0 (mg)&lt;/td&gt;
        &lt;/tr&gt;
        &lt;tr&gt;
            &lt;td&gt;Carbs&lt;/td&gt;
            &lt;td&gt;8.4 (mg)&lt;/td&gt;
        &lt;/tr&gt;
        &lt;tr&gt;
            &lt;td&gt;Dietary Fiber&lt;/td&gt;
            &lt;td&gt;0 (g)&lt;/td&gt;
        &lt;/tr&gt;
        &lt;tr&gt;
            &lt;td&gt;Sugar&lt;/td&gt;
            &lt;td&gt;8.4 (g)&lt;br /&gt;
            &amp;nbsp;&lt;/td&gt;
        &lt;/tr&gt;
    &lt;/tbody&gt;
&lt;/table&gt;
&amp;nbsp;
</code></pre>

<p>Ideally, I would like to be able to output to a CSV that has "Product Name" and Column 1 data in the header row, since it is the same for all of the tables. Then the data rows would be like:
 <code>"Product 1 Name, 8, 112, 0, 0.2, 8.8, 0, 8.8"</code></p>

<p>I know there is some manipulation that needs to be done to the data to get it to that point (to remove the size information).</p>

<p>Here is what I have so far that is starting to drive me crazy:</p>

<pre><code>import requests, bs4, urllib2, csv
from bs4 import BeautifulSoup
from collections import defaultdict


#Loop on URLs to get Nutritional Information from each one.
with open('NutritionalURLs.txt') as f:
    for line in f:
        r = requests.get('website' + line)
        soup=BeautifulSoup(r.text.encode('ascii','ignore'),"html.parser")

#TESTING
        with open('output.txt', 'w') as o:
            product_list = soup.find_all('b')
            product_list = soup.find_all('strong')
            print(product_list)
            table_list = soup.find_all('table')
            for tables in table_list:
                trs = tables.find_all('tr')
                for tr in trs:
                    tds = tr.find_all('td')[1:]
                    if tds:
                        facts = tds[0].find(text=True)
                        print(facts)
#                        o.write("Serving Size: %s, Calories: %s, Fat: %s, Sodium: %s, Carbs: %s, Dietary Fiber: %s, Sugar: %s\n" % \
#                            (facts[0].text, facts[1].text, facts[2].text, facts[3].text, facts[4].text, facts[5].text, facts[6].text)) 
</code></pre>

<p>This gives me an output like this:</p>

<pre><code>[&lt;strong&gt;Product 1 Name&lt;/strong&gt;, &lt;strong&gt;Product 2 Name&lt;/strong&gt;]
8 (fl. Oz.)
101 Calories
0 (g)
0.0 (mg)
0 (mg)
0 (g)
0 (g)
8 (fl. Oz.)
101 Calories
0 (g)
0.0 (mg)
0 (mg)
0 (g)
0 (g)
[]
</code></pre>

<p>Find the tables, then extract the text from the previous strong and take the second <em>td</em> from each <em>tr</em> splitting  the text once to remove the <code>(g)</code> etc..:</p>

<pre><code>from bs4 import BeautifulSoup

soup = BeautifulSoup(html)

for table in soup.find_all("table"):
    name = [table.find_previous("strong").text]
    amounts = [td.text.split(None, 1)[0] for  td in table.select("tr td + td")])
    print(name + amounts)
</code></pre>

<p>Which would give you:</p>

<pre><code>['Product 1 Name', '8', '122', '0', '0.2', '8.8', '0', '8.8']
['Product 2 Name', '8', '134', '0', '0.0', '8.4', '0', '8.4']
</code></pre>

<p><em>select("tr td + td")</em> uses a css selector to get the second <em>td</em> from each <em>tr/row</em>,</p>

<p>Or using <em>find_all</em> and slicing would look like:</p>

<pre><code>for table in soup.find_all("table"):
    name = [table.find_previous("strong").text]
    amounts = [td.find_all("td")[1].text.split(None, 1)[0] for  td in table.find_all("tr")]
    print(name + amounts)
</code></pre>

<p>Since it is not always a strong but sometimes a bold tag you want, just look for the strong first and fall back to the bold:</p>

<pre><code>from bs4 import BeautifulSoup
import requests
html = requests.get("http://beamsuntory.desk.com/customer/en/portal/articles/1676001-nutrition-information-cruzan").content
soup = BeautifulSoup(html, "html.parser")
for table in soup.select("div.article-content table"):
    name = table.find_previous("strong") or table.find_previous("b")
    amounts = [td.text.split(None, 1)[0] for  td in table.select("tr td + td")]
    print([name.text] + amounts)
</code></pre>

<p>If <em>table.find_previous("strong")</em> finds nothing it will be None so the or will be executed and name will be set to <em>table.find_previous("b")</em>.</p>

<p>Now it will work for both:</p>

<pre><code>In [12]: html = requests.get("http://beamsuntory.desk.com/customer/en/portal/articles/1676001-nutrition-information-cruzan").content

In [13]: soup = BeautifulSoup(html, "html.parser")

In [14]: for table in soup.select("div.article-content table"):
   ....:         name = table.find_previous("strong") or table.find_previous("b")
   ....:         amounts = [td.text.split(None, 1)[0] for  td in table.select("tr td + td")]
   ....:         print([name.text] + amounts)
   ....:     
[u'Cruzan Banana Flavored Rum 42 proof', u'1.5', u'79', u'0', u'0.0', u'6.5', u'0', u'6.5']
[u'Cruzan Banana Flavored Rum 55 proof', u'1.5', u'95', u'0', u'0.0', u'6.5', u'0', u'6.5']
[u'Cruzan Black Cherry Flavored Rum 42 proof', u'1.5', u'80', u'0', u'0.0', u'6.9', u'0', u'6.9']
[u'Cruzan Citrus Flavored Rum 42 proof', u'1.5', u'99', u'0', u'0.0', u'2.8', u'0', u'2.6']
[u'Cruzan Coconut Flavored Rum 42 proof', u'1.5', u'78', u'0', u'0.1', u'6.9', u'0', u'6.5']
[u'Cruzan Coconut Flavored Rum 55 proof', u'1.5', u'95', u'0', u'0.1', u'6.1', u'0', u'0']
[u'Cruzan Guaza Flavored Rum 42 proof', u'1.5', u'78', u'0', u'0.1', u'6.5', u'0', u'6.5']
[u'Cruzan Key Lime Flavored Rum 42 proof', u'1.5', u'81', u'0', u'0.0', u'8.1', u'0', u'6']
[u'Cruzan Mango Flavored Rum 42 proof', u'1.5', u'85', u'0', u'0.0', u'8.5', u'0', u'8.5']
[u'Cruzan Mango Flavored Rum 55 proof', u'1.5', u'101', u'0', u'0.0', u'8.5', u'0', u'8.5']
[u'Cruzan Orange Flavored Rum 42 proof', u'1.5', u'76.77', u'0', u'0', u'6.4', u'0', u'6.4']
[u'Cruzan Passion Fruit Flavored Rum 42 proof', u'1.5', u'77', u'0', u'0.0', u'6.3', u'0', u'6.3']
[u'Cruzan Pineapple Flavored Rum 42 proof', u'1.5', u'78', u'0', u'0.0', u'6.5', u'0', u'6.5']
[u'Cruzan Pineapple Flavored Rum 55 proof', u'1.5', u'94', u'0', u'0.0', u'6.5', u'0', u'6.5']
[u'Cruzan Raspberry Flavored Rum 42 proof', u'1.5', u'92', u'0', u'0.0', u'10.1', u'0', u'10.1']
[u'Cruzan Raspberry Flavored Rum 55 proof', u'1.5', u'108', u'0', u'0.0', u'10.1', u'0', u'10.1']
[u'Cruzan Strawberry Flavored Rum 42 proof', u'1.5', u'76', u'0', u'0.0', u'6.1', u'0', u'6']
[u'Cruzan Vanilla Flavored Rum 42 proof', u'1.5', u'78', u'0', u'0.0', u'6.5', u'0', u'6.5']
[u'Cruzan Vanilla Flavored Rum 55 proof', u'1.5', u'94', u'0', u'0.0', u'6.5', u'0', u'6.5']
[u'Cruzan Estate Dark Rum 80 proof', u'1.5', u'101', u'0', u'0.0', u'0', u'0', u'0']
[u'Cruzan Estate Light Rum 80 proof', u'1.5', u'101', u'0', u'0.0', u'0', u'0', u'0']
[u'Cruzan Estate Single Barrel Rum 80 proof', u'1.5', u'99', u'0', u'0.0', u'0.9', u'0', u'0.9']
</code></pre>

<p>And the bold:</p>

<pre><code>In [20]: html = requests.get("http://beamsuntory.desk.com/customer/en/portal/articles/1790163-midori-nutrition-information").content

In [21]: soup = BeautifulSoup(html, "html.parser")

In [22]: for table in soup.select("div.article-content table"):
   ....:         name = table.find_previous("strong") or table.find_previous("b")
   ....:         amounts = [td.text.split(None, 1)[0] for  td in table.select("tr td + td")]
   ....:         print([name.text] + amounts)
   ....:     
[u'Midori', u'1.0', u'62.1', u'0', u'0.3', u'7.5', u'0', u'7.0']
</code></pre>

python html web-scraping beautifulsoup