Match URLs by file path and GET parameters (but not their values)
<p>How can I check if any of my list of URLs match the given <code>url</code>? I need URLs to match only if all GET parameter names (not their values) and the path are the same. For example, I have this list:</p>

<pre><code>links = [
  "http://example.com/page.php?param1=111&amp;param2=222",
  "http://example.com/page2.php?param1=111&amp;param2=222",
  "http://example.com/page2.php?param1=111&amp;param2=222&amp;someParameterN=NumberN"
  ]

url = "http://example.com/page2.php?param1=NOT111&amp;param2=NOT222"
</code></pre>

<p>This example is <code>True</code> because <code>url</code> matches <code>links[1]</code>. But how to match it in the most efficient way? I don't know what <code>url</code> will looks like.</p>

<p>Python's standard library comes with a package for parsing urls: <a href="https://docs.python.org/3/library/urllib.parse.html" rel="nofollow"><code>urllib.parse</code></a>.  Don't try to write your own regexes for this... <em>especially</em> if you haven't considered all the weird things that are legal parts of a URL.</p>

<p>I suggest something like below.  <code>is_url_in_list</code> is the question you want answered.  It calls <code>url_file_and_params</code> to break the URl into the file path and a set of query parameters.  <code>url_file_and_params</code> calls <code>url_params_from_quoted_query</code> to build the set of parameter names.</p>

<pre class="lang-python3 prettyprint-override"><code>#!/usr/bin/env python3

from urllib.parse import parse_qs
from urllib.parse import urlsplit


def url_params_from_quoted_query(query_string):
    # An empty query string would make parse_qs raise a ValueError.
    if '' == query_string:
        return set()
    params_and_values = parse_qs(
      query_string,
      keep_blank_values=True,
      strict_parsing=True,
      )
    params = set(params_and_values)
    return params


def url_file_and_params(url):
    parts = urlsplit(url)
    url_file = parts[2]
    quoted_query = parts[3]
    url_params = url_params_from_quoted_query(quoted_query)
    return url_file, url_params


def is_url_in_list(url_target, url_list):
    target_file, target_params = url_file_and_params(url_target)
    for url in url_list:
        url_file, url_params = url_file_and_params(url)
        if url_file == target_file and url_params == target_params:
            return True
    return False


def main():
    links = [
      "http://example.com/page.php?param1=111&amp;param2=222",
      "http://example.com/page2.php?param1=111&amp;param2=222",
      "http://example.com/page2.php?param1=&amp;param2=222",
      "http://example.com/page2.php",
      "http://example.com/page2.php?param1=111&amp;param2=222&amp;someParameterN=NumberN"
      ]

    url = "http://example.com/page2.php?param1=NOT111&amp;param2=NOT222"
    print(is_url_in_list(url, links))
    return


if "__main__" == __name__:
    main()
</code></pre>

<p>One assumption this code makes is that your URLs are already UTF-8 strings with correctly percent-encoded query strings.  If not, you might need to use <a href="https://docs.python.org/3/library/urllib.parse.html#urllib.parse.quote" rel="nofollow"><code>quote</code></a> or <a href="https://docs.python.org/3/library/urllib.parse.html#urllib.parse.quote_from_bytes" rel="nofollow"><code>quote_from_bytes</code></a> before feeding them to <code>is_url_in_list</code>.</p>

python regex list url match