Match URLs by file path and GET parameters (but not their values)
<p>How can I check if any of my list of URLs match the given <code>url</code>? I need URLs to match only if all GET parameter names (not their values) and the path are the same. For example, I have this list:</p>

<pre><code>links = [
  "http://example.com/page.php?param1=111&amp;param2=222",
  "http://example.com/page2.php?param1=111&amp;param2=222",
  "http://example.com/page2.php?param1=111&amp;param2=222&amp;someParameterN=NumberN"
  ]

url = "http://example.com/page2.php?param1=NOT111&amp;param2=NOT222"
</code></pre>

<p>This example is <code>True</code> because <code>url</code> matches <code>links[1]</code>. But how to match it in the most efficient way? I don't know what <code>url</code> will looks like.</p>

<p>You ideally want to use python's urlparse library.
Parse your url like so:</p>

<pre><code>import urlparse
url = "http://example.com/page2.php?param1=NOT111&amp;param2=NOT222"
parsed_url = urlparse.urlparse(url)
urlparse.parse_qs(parsed_url.query).keys()
</code></pre>

<p>Then you create a datastructure which looks something like this:</p>

<pre><code>seen_pages = set() # Stores all pages you've already seen.
</code></pre>

<p>And then all your pages to it like so:</p>

<pre><code>for page in list_of_pages:
    parsed_url = urlparse.urlparse(page)
    current_page = (parsed_url.path, frozenset(urlparse.parse_qs(parsed_url.query).keys())
    seen_pages.add(current_page)
</code></pre>

<p>This stores all your pages in the form: <code>tuple(link, set(param1,param2))</code> in a set.</p>

<p>To look up if you've already visited the page, with those exact parameters, simply create the <code>current_page</code> structure again and look it up in the set.
Look up and addition to a set is an <code>O(1)</code> operation, that is, it is as fast as you can get.</p>

python regex list url match