Grouping by set . To group files with similar headers python
<p>I have derived an output from a directory structure which has a lot many csv files. The headers of these files are manually created and randomly placed. I have to get all those files which have similar headers together.</p>

<pre><code>/A/B/C/D~b1.csv.0   Delim:,
"First Name"    "Last Name" Company EMAIL   Phone   Fax "SIC CODE"
/A/B/C/D~b2.csv.0   Delim:,
"First Name"    "Last Name" Phone   Fax "SIC CODE"  Company EMAIL
/A/B/C/D~b3.csv.0   Delim:,
"First Name"    "Last Name" Company EMAIL   Fax "SIC CODE"  Phone
/A/B/C/D~b4.csv.0   Delim:,
"First Name"    "Last Name" Company EMAIL   Phone   Fax "SIC CODE"
/A/B/C/D~c1.csv.0   Delim:,
"Business Type" "Main Markets"  Establised  "No Of Employees"   Category    "Company Name"  "Contact Person"    Designation Address "Pin Code"  "Telephone no"  "Fax No"    Country Website Email
/A/B/C/D~c2.csv.0   Delim:,
"Business Type" "Main Markets"  Establised  "No Of Employees"   Category    "Company Name"  "Contact Person"    Designation Address "Pin Code"  "Telephone no"  "Fax No"    Country Website Email
/A/B/C/D~c3.csv.0   Delim:,
"Business Type" "Main Markets"  Country Website Email   Establised  "No Of Employees"   Category    "Company Name"  "Contact Person"    Designation Address "Pin Code"  "Telephone no"  "Fax No"
</code></pre>

<p>The first part <code>/A/B/C/D</code> is the directory structure followed by a <code>~</code> followed by the Delimiter <code>Delim:,</code> required to parse the file. The next line is the header which was fetched from the file <code>"First Name"    "Last Name" Company EMAIL   Phone   Fax "SIC CODE"</code></p>

<p>I tried to create a sample code to group similar headers together, something as follows which I know wouldn't have worked:</p>

<pre><code>&gt;&gt;&gt; li = [('abc', set(['a', 'c', 'b'])), ('def', set(['e', 'd', 'f'])), ('ghi', set(['i', 'h', 'g'])), ('jkl', set(['k', 'j', 'l'])), ('mno', set(['m', 'o', 'n'])), ('pqr', set(['q', 'p', 'r'])), ('stu', set(['s', 'u', 't'])), ('vwx', set(['x', 'w', 'v'])), ('ABC', set(['a', 'c', 'b'])), ('DEF', set(['e', 'd', 'f'])), ('GHI', set(['i', 'h', 'g'])), ('JKL', set(['k', 'j', 'l'])), ('MNO', set(['m', 'o', 'n'])), ('PQR', set(['q', 'p', 'r'])), ('STU', set(['s', 'u', 't'])), ('VWX', set(['x', 'w', 'v']))]
&gt;&gt;&gt; for key, group in groupby(li, lambda x: x[1]):
...     for l in group:
...         print "%s %s." % (l[1], l[0])
</code></pre>

<p>How can I group the sets together.
Any help figuring out how I could group similar header files is appreciated.</p>

<p>The following approach works by taking each of your CSV headers and converting them into a list of column entries. These are then sorted and converted to a tuple. This is then used as the key for a default dictionary. Each matching entry is appended to the list along with the original column ordering. </p>

<p>The result is a dictionary which groups together CSV files containing the same column entries. If column entries are not case sensitive, the tuple entries could be converted to lowercase before being used as the key.</p>

<pre><code>from collections import defaultdict
import csv
from StringIO import StringIO

csv_groups = defaultdict(list)

entries = [
    ["/A/B/C/D~b1.csv.0", "Delim:,", '"First Name"    "Last Name" Company EMAIL   Phone   Fax "SIC CODE"'],
    ["/A/B/C/D~b2.csv.0", "Delim:,", '"First Name"    "Last Name" Phone   Fax "SIC CODE"  Company EMAIL'],
    ["/A/B/C/D~b3.csv.0", "Delim:,", '"First Name"    "Last Name" Company EMAIL   Fax "SIC CODE"  Phone'],
    ["/A/B/C/D~b4.csv.0", "Delim:,", '"First Name"    "Last Name" Company EMAIL   Phone   Fax "SIC CODE"'],
    ["/A/B/C/D~c1.csv.0", "Delim:,", '"Business Type" "Main Markets"  Establised  "No Of Employees"   Category    "Company Name"  "Contact Person"    Designation Address "Pin Code"  "Telephone no"  "Fax No"    Country Website Email'],
    ["/A/B/C/D~c2.csv.0", "Delim:,", '"Business Type" "Main Markets"  Establised  "No Of Employees"   Category    "Company Name"  "Contact Person"    Designation Address "Pin Code"  "Telephone no"  "Fax No"    Country Website Email'],
    ["/A/B/C/D~c3.csv.0", "Delim:,", '"Business Type" "Main Markets"  Country Website Email   Establised  "No Of Employees"   Category    "Company Name"  "Contact Person"    Designation Address "Pin Code"  "Telephone no"  "Fax No"']
]

for folder, delim, header in entries:
    cols = tuple(sorted(list(csv.reader(StringIO(header), delimiter=' ', skipinitialspace=True))[0]))
    csv_groups[cols].append((folder, header))

for csv_type, folders in csv_groups.iteritems():
    print csv_type

    for folder in folders:
        print "  ", folder
</code></pre>

<p>This would give you the following grouping based on your data:</p>

<pre class="lang-none prettyprint-override"><code>('Address', 'Business Type', 'Category', 'Company Name', 'Contact Person', 'Country', 'Designation', 'Email', 'Establised', 'Fax No', 'Main Markets', 'No Of Employees', 'Pin Code', 'Telephone no', 'Website')
   ('/A/B/C/D~c1.csv.0', '"Business Type" "Main Markets"  Establised  "No Of Employees"   Category    "Company Name"  "Contact Person"    Designation Address "Pin Code"  "Telephone no"  "Fax No"    Country Website Email')
   ('/A/B/C/D~c2.csv.0', '"Business Type" "Main Markets"  Establised  "No Of Employees"   Category    "Company Name"  "Contact Person"    Designation Address "Pin Code"  "Telephone no"  "Fax No"    Country Website Email')
   ('/A/B/C/D~c3.csv.0', '"Business Type" "Main Markets"  Country Website Email   Establised  "No Of Employees"   Category    "Company Name"  "Contact Person"    Designation Address "Pin Code"  "Telephone no"  "Fax No"')
('Company', 'EMAIL', 'Fax', 'First Name', 'Last Name', 'Phone', 'SIC CODE')
   ('/A/B/C/D~b1.csv.0', '"First Name"    "Last Name" Company EMAIL   Phone   Fax "SIC CODE"')
   ('/A/B/C/D~b2.csv.0', '"First Name"    "Last Name" Phone   Fax "SIC CODE"  Company EMAIL')
   ('/A/B/C/D~b3.csv.0', '"First Name"    "Last Name" Company EMAIL   Fax "SIC CODE"  Phone')
   ('/A/B/C/D~b4.csv.0', '"First Name"    "Last Name" Company EMAIL   Phone   Fax "SIC CODE"')          
</code></pre>

python python-2.7