Checking if element in list by substring
<p>I have a list of urls (<code>unicode</code>), and there is a lot of repetition.
For example, urls <code>http://www.myurlnumber1.com</code> and <code>http://www.myurlnumber1.com/foo+%bar%baz%qux</code> lead to the same place.</p>

<p>So I need to weed out all of those duplicates.</p>

<p>My first idea was to check if the element's substring is in the list, like so:</p>

<pre><code>for url in list:
    if url[:30] not in list:
        print(url)
</code></pre>

<p>However, it tries to mach literal <code>url[:30]</code> to a list element and obviously returns all of them, since there is no element that exactly matches <code>url[:30]</code>. </p>

<p>Is there an easy way to solve this problem?</p>

<p>EDIT:</p>

<p>Often the host and path in the urls stays the same, but the parameters are different. For my purposes, a url with the same hostname and path, but different parameters are still the same url and constitute a duplicate.</p>

<p>You can try adding another for loop, if you are fine with that.
Something like:</p>

<pre><code>for url in list:  
    for i in range(len(list)):  
      if url[:30] not in list[i]:  
          print(url)  
</code></pre>

<p>That will compare every word with every other word to check for sameness. That's just an example, I'm sure you could make it more robust.</p>

python list