Checking if element in list by substring
<p>I have a list of urls (<code>unicode</code>), and there is a lot of repetition.
For example, urls <code>http://www.myurlnumber1.com</code> and <code>http://www.myurlnumber1.com/foo+%bar%baz%qux</code> lead to the same place.</p>

<p>So I need to weed out all of those duplicates.</p>

<p>My first idea was to check if the element's substring is in the list, like so:</p>

<pre><code>for url in list:
    if url[:30] not in list:
        print(url)
</code></pre>

<p>However, it tries to mach literal <code>url[:30]</code> to a list element and obviously returns all of them, since there is no element that exactly matches <code>url[:30]</code>. </p>

<p>Is there an easy way to solve this problem?</p>

<p>EDIT:</p>

<p>Often the host and path in the urls stays the same, but the parameters are different. For my purposes, a url with the same hostname and path, but different parameters are still the same url and constitute a duplicate.</p>

<p>If you consider any netloc's to be the same you can parse with <a href="https://docs.python.org/3/library/urllib.parse.html#urllib.parse.urlparse" rel="nofollow"><code>urllib.parse</code></a></p>

<pre><code>from urllib.parse import  urlparse # python2 from urlparse import  urlparse 

u = "http://www.myurlnumber1.com/foo+%bar%baz%qux"

print(urlparse(u).netloc)
</code></pre>

<p>Which would give you:</p>

<pre><code>www.myurlnumber1.com
</code></pre>

<p>So to get unique netlocs you could do something like:</p>

<pre><code>unique  = {urlparse(u).netloc for u in urls}
</code></pre>

<p>If you wanted to keep the url scheme:</p>

<pre><code>urls  = ["http://www.myurlnumber1.com/foo+%bar%baz%qux", "http://www.myurlnumber1.com"]

unique = {"{}://{}".format(u.scheme, u.netloc) for u in map(urlparse, urls)}
print(unique)
</code></pre>

<p>Presuming they all have schemes and you don't have http and https for the same netloc and consider them to be the same.</p>

<p>If you also want to add the path:</p>

<pre><code>unique = {u.netloc, u.path) for u in map(urlparse, urls)}
</code></pre>

<p>The table of attributes is listed in the docs:</p>

<pre><code>Attribute   Index   Value   Value if not present
scheme  0   URL scheme specifier    scheme parameter
netloc  1   Network location part   empty string
path    2   Hierarchical path   empty string
params  3   Parameters for last path element    empty string
query   4   Query component empty string
fragment    5   Fragment identifier empty string
username        User name   None
password        Password    None
hostname        Host name (lower case)  None
port        Port number as integer, if present  None
</code></pre>

<p>You just need to use whatever you consider to be the unique parts.</p>

<pre><code>In [1]: from urllib.parse import  urlparse

In [2]: urls = ["http://www.url.com/foo-bar", "http://www.url.com/foo-bar?t=baz", "www.url.com/baz-qux",  "www.url.com/foo-bar?t=baz"]


In [3]: unique = {"".join((u.netloc, u.path)) for u in map(urlparse, urls)}

In [4]: 

In [4]: print(unique)
{'www.url.com/baz-qux', 'www.url.com/foo-bar'}
</code></pre>

python list