parse table using beautifulsoup in python
<p>I want to traverse through each row and capture values of td.text. However problem here is table does not have class. and all the td got same class name. I want to traverse through each row and want following output:</p>

<p>1st row)"AMERICANS SOCCER CLUB","B11EB - AMERICANS-B11EB-WARZALA","Cameron Coya","Player 228004","2016-09-10","player persistently infringes the laws of the game","C" (new line)</p>

<p>2nd row) "AVIATORS SOCCER CLUB","G12DB - AVIATORS-G12DB-REYNGOUDT","Saskia Reyes","Player 224463","2016-09-11","player/sub guilty of unsporting behavior"," C" (new line)</p>

<pre><code>&lt;div style="overflow:auto; border:1px #cccccc solid;"&gt;
&lt;table cellspacing="0" cellpadding="3" align="left" border="0" width="100%"&gt;
    &lt;tbody&gt;
        &lt;tr class="tblHeading"&gt;
            &lt;td colspan="7"&gt;AMERICANS SOCCER CLUB&lt;/td&gt;
        &lt;/tr&gt;
        &lt;tr bgcolor="#CCE4F1"&gt;
            &lt;td colspan="7"&gt;B11EB - AMERICANS-B11EB-WARZALA&lt;/td&gt; 
        &lt;/tr&gt;
        &lt;tr bgcolor="#FFFFFF"&gt;
            &lt;td width="19%" class="tdUnderLine"&gt; &amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; Cameron Coya                                       &lt;/td&gt;
            &lt;td width="19%" class="tdUnderLine"&gt;
                Rozel, Max
            &lt;/td&gt;
            &lt;td width="06%" class="tdUnderLine"&gt; 
            09-11-2016
            &lt;/td&gt;
            &lt;td width="05%" class="tdUnderLine" align="center"&gt;         
                &lt;a href="http://www.ncsanj.com/gameRefReportPrint.cfm?gid=228004" target="_blank"&gt;228004&lt;/a&gt;    
            &lt;/td&gt;
            &lt;td width="16%" class="tdUnderLine" align="center"&gt; 
                09/10/16 02:15 PM   
            &lt;/td&gt;
            &lt;td width="30%" class="tdUnderLine"&gt;                player persistently infringes the laws of the game   &lt;/td&gt;
            &lt;td class="tdUnderLine"&gt;                Cautioned    &lt;/td&gt;
        &lt;/tr&gt;
        &lt;tr class="tblHeading"&gt;
            &lt;td colspan="7"&gt;AVIATORS SOCCER CLUB&lt;/td&gt;
        &lt;/tr&gt;
        &lt;tr bgcolor="#CCE4F1"&gt;
            &lt;td colspan="7"&gt;G12DB - AVIATORS-G12DB-REYNGOUDT&lt;/td&gt; 
        &lt;/tr&gt;
        &lt;tr bgcolor="#FBFBFB"&gt;
            &lt;td width="19%" class="tdUnderLine"&gt; &amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; Saskia Reyes                                       &lt;/td&gt;
            &lt;td width="19%" class="tdUnderLine"&gt;
                HollaenderNardelli, Eric
            &lt;/td&gt;
            &lt;td width="06%" class="tdUnderLine"&gt; 
            09-11-2016
            &lt;/td&gt;
            &lt;td width="05%" class="tdUnderLine" align="center"&gt;         

                &lt;a href="http://www.ncsanj.com/gameRefReportPrint.cfm?gid=224463" target="_blank"&gt;224463&lt;/a&gt;    
            &lt;/td&gt;
            &lt;td width="16%" class="tdUnderLine" align="center"&gt; 
                09/11/16 06:45 PM   
            &lt;/td&gt;
            &lt;td width="30%" class="tdUnderLine"&gt;                player/sub guilty of unsporting behavior     &lt;/td&gt;
            &lt;td class="tdUnderLine"&gt;                Cautioned    &lt;/td&gt;
        &lt;/tr&gt;
        &lt;tr class="tblHeading"&gt;
            &lt;td colspan="7"&gt;BERGENFIELD SOCCER CLUB&lt;/td&gt;
        &lt;/tr&gt;
        &lt;tr bgcolor="#CCE4F1"&gt;
            &lt;td colspan="7"&gt;B11CW - BERGENFIELD-B11CW-NARVAEZ&lt;/td&gt; 
        &lt;/tr&gt;
        &lt;tr bgcolor="#FFFFFF"&gt;
            &lt;td width="19%" class="tdUnderLine"&gt; &amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; Christian Latorre                                  &lt;/td&gt;
            &lt;td width="19%" class="tdUnderLine"&gt;
                Coyle, Kevin
            &lt;/td&gt;
            &lt;td width="06%" class="tdUnderLine"&gt; 
            09-10-2016
            &lt;/td&gt;
            &lt;td width="05%" class="tdUnderLine" align="center"&gt;         

                &lt;a href="http://www.ncsanj.com/gameRefReportPrint.cfm?gid=226294" target="_blank"&gt;226294&lt;/a&gt;    
            &lt;/td&gt;
            &lt;td width="16%" class="tdUnderLine" align="center"&gt; 

                09/10/16 11:00 AM   

            &lt;/td&gt;
            &lt;td width="30%" class="tdUnderLine"&gt;                player persistently infringes the laws of the game   &lt;/td&gt;
            &lt;td class="tdUnderLine"&gt;                Cautioned    &lt;/td&gt;
        &lt;/tr&gt;
</code></pre>

<p>I tried with following code. </p>

<pre><code>import requests
from bs4 import BeautifulSoup
import re
try:
    import urllib.request as urllib2
except ImportError:
    import urllib2

url = r"G:\Freelancer\NC Soccer\Northern Counties Soccer Association Â©.html"
page = open(url, encoding="utf8")
soup = BeautifulSoup(page.read(),"html.parser")

#tableList = soup.findAll("table")

for tr in soup.find_all("tr"):
    for td in tr.find_all("td"):
        print(td.text.strip())
</code></pre>

<p>but it is obvious that it will return text form all td and I will not able to identify particular column name or will not able to determine start of new record. I want to know </p>

<p>1) how to identify each column(because class name is same) and there are headings as well (I will appreciate if you provide code for that) </p>

<p>2) how to identify new record in such structure</p>

<pre><code>from __future__ import print_function
import re
import datetime
from bs4 import BeautifulSoup

soup = ""
with open("/tmp/a.html") as page:
   soup = BeautifulSoup(page.read(),"html.parser")

table = soup.find('div', {'style': 'overflow:auto; border:1px #cccccc solid;'}).find('table')

trs = table.find_all('tr')

table_dict = {}
game = ""
section = ""

for tr in trs:
    if tr.has_attr('class'):
        game = tr.text.strip('\n')
    if tr.has_attr('bgcolor'):
        if tr['bgcolor'] == '#CCE4F1':
            section = tr.text.strip('\n')
        else:
            tds = tr.find_all('td')
            extracted_text = [re.sub(r'([^\x00-\x7F])+','', td.text) for td in tds]
            extracted_text = [x.strip() for x in extracted_text]
            extracted_text = list(filter(lambda x: len(x) &gt; 2, extracted_text))
            extracted_text.pop(1)
            extracted_text[2] = "Player " + extracted_text[2]
            extracted_text[3] = datetime.datetime.strptime(extracted_text[3], '%m/%d/%y %I:%M %p').strftime("%Y-%m-%d")
            extracted_text = ['"' + x + '"' for x in [game, section] + extracted_text]
            print(','.join(extracted_text))
</code></pre>

<p>And when run:</p>

<pre><code>$ python a.py

"AMERICANS SOCCER CLUB","B11EB - AMERICANS-B11EB-WARZALA","Cameron Coya","Player 228004","2016-09-10","player persistently infringes the laws of the game","C"
"AVIATORS SOCCER CLUB","G12DB - AVIATORS-G12DB-REYNGOUDT","Saskia Reyes","Player 224463","2016-09-11","player/sub guilty of unsporting behavior","C"
"BERGENFIELD SOCCER CLUB","B11CW - BERGENFIELD-B11CW-NARVAEZ","Christian Latorre","Player 226294","2016-09-10","player persistently infringes the laws of the game","C"
</code></pre>

<p>Based on further conversation with the OP, the input was <a href="https://paste.fedoraproject.org/428111/87928814/raw/" rel="nofollow">https://paste.fedoraproject.org/428111/87928814/raw/</a> and the output after running the above code is: <a href="https://paste.fedoraproject.org/428110/38792211/raw/" rel="nofollow">https://paste.fedoraproject.org/428110/38792211/raw/</a></p>

python web-scraping beautifulsoup