Scraping values from a webpage table
<p>I want to create a python dictionary of color names to background color from this <a href="http://people.csail.mit.edu/jaffer/Color/M.htm" rel="nofollow">color dictionary</a>.</p>

<p>What is the best way to access the color name strings and the background color hex values? I want to create a mapping for color name --> hex values, where 1 color name maps to 1 or more hex values.</p>

<p>The following is my code:</p>

<pre><code>import requests
from bs4 import BeautifulSoup

page = requests.get('http://people.csail.mit.edu/jaffer/Color/M.htm')
soup = BeautifulSoup(page.text)
</code></pre>

<p>I'm not sure how to specify what to scrape from the table. I've tried the following to get a format that's useful:</p>

<pre><code>soup.td
&lt;td nowrap="" width="175*"&gt;abbey&lt;/td&gt;


soup.get_text()
"(M)\n td { padding: 0 10px; }  \n\n(M) Dictionary of Color  Maerz and Paul, Dictionary of Color, 1st ed.   \n\nabbey207\nabsinthe [green]  120\nabsinthe yellow105\nacacia101102\nacademy blue173\nacajou43\nacanthe95\nacier109\nackermann's green137\naconite violet223....
.............\nyolk yellow84\nyosemite76\nyucatan5474\nyucca150\nyu chi146\nyvette violet228\n\nzaffre blue  179182\nzanzibar47\nzedoary wash71\nzenith [blue]  199203\nzephyr78\nzinc233265\nzinc green136\nzinc orange5053\nzinc yellow84\nzinnia15\nzulu47\nzuni brown58\n\n"


soup.select('tr td')
[...
&lt;td nowrap="" width="175*"&gt;burnt russet&lt;/td&gt;,
 &lt;td style="background-color:#722F37; color:#FFF" title="16"&gt;16&lt;/td&gt;,
 &lt;td style="background-color:#79443B; color:#FFF" title="43"&gt;43
 &lt;/td&gt;,
 &lt;td nowrap="" width="175*"&gt;burnt sienna&lt;/td&gt;,
 &lt;td style="background-color:#9E4732; color:#FFF" title="38"&gt;38
 &lt;/td&gt;,
 ...]
</code></pre>

<p>EDIT:
I want to scrape the strings in the td elements e.g "burnt russet" as the color and the string (hex component) in the following td elements where the "style" attribute is specified as the background color.</p>

<p>I want the dictionary to look as follows:</p>

<pre><code>color_map = {'burnt russet': [#722F37, #79443B], 'burnt sienna': [#9E4732]}
</code></pre>

<p>Just look for the tds with nowrap, extract the text and get the following siblings td's <em>style</em> attribute:</p>

<pre><code>import requests
from bs4 import BeautifulSoup

page = requests.get('http://people.csail.mit.edu/jaffer/Color/M.htm')
soup = BeautifulSoup(page.content)
for td in soup.select("td[nowrap]"):
    print(td.text, [sib["style"] for sib in td.find_next_siblings("td")])
</code></pre>

<p>A snippet of the output:</p>

<pre><code> (u'abbey', ['background-color:#604E97; color:#FFF'])
(u'absinthe [green]  ', ['background-color:#8A9A5B'])
(u'absinthe yellow', ['background-color:#B9B57D'])
(u'acacia', ['background-color:#EAE679', 'background-color:#B9B459'])
(u'academy blue', ['background-color:#367588'])
(u'acajou', ['background-color:#79443B; color:#FFF'])
(u'acanthe', ['background-color:#6C541E; color:#FFF'])
(u'acier', ['background-color:#8C8767'])
(u"ackermann's green", ['background-color:#355E3B; color:#FFF'])
(u'aconite violet', ['background-color:#86608E'])
(u'acorn', ['background-color:#7E6D5A; color:#FFF'])
(u'adamia', ['background-color:#563C5C; color:#FFF'])
(u'adelaide', ['background-color:#32174D; color:#FFF'])
</code></pre>

<p>If you just want the hex values you can split the style text on <code>"; "</code> then split the sub strings on <code>:</code>:</p>

<pre><code>page = requests.get('http://people.csail.mit.edu/jaffer/Color/M.htm')
soup = BeautifulSoup(page.content)
d = {}
for td in soup.select("td[nowrap]"):
    cols = td.find_next_siblings("td")
    d[td.text] = [st.split(":", 1)[-1] for sib in cols for st in sib["style"].split("; ")]

print(d)
</code></pre>

<p>That will give you a dioct like:</p>

<pre><code>{u'moonlight ': ['#FAD6A5', '#BFB8A5'], u'honey bird': ['#239EBA'], u'monte carlo ': ['#007A74', '#317873'],...............
</code></pre>

<p>You will need to use either <code>lxml</code> or <code>html5lib</code> as the parser to handle the broken html. I presume you are using one of them as if not you would not get the output you do.</p>

python html beautifulsoup