Python: extract text request from url
<p>I try to extract users requests from url. I try to search answer, but I only find how to parse string.
But I have a problem, that I should identify a lot of urls with request and when I try to get string with attribute, attributes with text are different.
I mean when I try</p>

<pre><code>pat = re.compile(r"\?\w+=(.*)")
search = ['yandex.ru/search', 'youtube.com/results', 'google.com/search', 'google.ru/search', 'go.mail.ru/search', 'search.yahoo.com/search', 'market.yandex.ru/search', 'bing.com/search']
for i in urls:
    u = re.findall(pat, i)
    if any(ext in i for ext in search):
        if len(u) &gt; 0:
            str = urllib.unquote(u[0])
            print str
            print {k: [s for s in v] for k, v in parse_qs(str).items()}
</code></pre>

<p>And it looks like </p>

<pre><code>chromesearch&amp;clid=2196598&amp;text=ÐºÐ¾ÑÐ¾Ð»ÐµÐ²Ñ ÐºÑÐ¸ÐºÐ° ÑÐ¼Ð¾ÑÑÐµÑÑ Ð¾Ð½Ð»Ð°Ð¹Ð½&amp;lr=213&amp;redircnt=1467230336.1
{'text': ['\xd0\xba\xd0\xbe\xd1\x80\xd0\xbe\xd0\xbb\xd0\xb5\xd0\xb2\xd1\x8b \xd0\xba\xd1\x80\xd0\xb8\xd0\xba\xd0\xb0 \xd1\x81\xd0\xbc\xd0\xbe\xd1\x82\xd1\x80\xd0\xb5\xd1\x82\xd1\x8c \xd0\xbe\xd0\xbd\xd0\xbb\xd0\xb0\xd0\xb9\xd0\xbd'], 'clid': ['2196598'], 'lr': ['213'], 'redircnt': ['1467230336.1']}
Ð¼Ð¸Ð½Ð¸Ð¼Ð°Ð»Ð¸ÑÑÐ¸ÑÐ½ÑÐ¹+ÑÑÐ¸Ð»Ñ&amp;newwindow=1&amp;biw=1280&amp;bih=909&amp;source=lnms&amp;tbm=isch&amp;sa=X&amp;ved=0ahUKEwikhI2M_s3NAhXBBiwKHfbEBEEQ_AUIBigB#imgrc=Er7qLiHoEGPIGM:
{'bih': ['909'], 'newwindow': ['1'], 'source': ['lnms'], 'ved': ['0ahUKEwikhI2M_s3NAhXBBiwKHfbEBEEQ_AUIBigB#imgrc=Er7qLiHoEGPIGM:'], 'tbm': ['isch'], 'biw': ['1280'], 'sa': ['X']}
Ð¼Ð¸Ð½Ð¸Ð¼Ð°Ð»Ð¸ÑÑÐ¸ÑÐ½ÑÐ¹+ÑÑÐ¸Ð»Ñ&amp;newwindow=1&amp;biw=1280&amp;bih=909&amp;source=lnms&amp;tbm=isch&amp;sa=X&amp;ved=0ahUKEwikhI2M_s3NAhXBBiwKHfbEBEEQ_AUIBigB#imgrc=Er7qLiHoEGPIGM:
{'bih': ['909'], 'newwindow': ['1'], 'source': ['lnms'], 'ved': ['0ahUKEwikhI2M_s3NAhXBBiwKHfbEBEEQ_AUIBigB#imgrc=Er7qLiHoEGPIGM:'], 'tbm': ['isch'], 'biw': ['1280'], 'sa': ['X']}
rjulf+ddjlbim+ytdthysq+gby+rjl+d+,fyrjvfn&amp;ie=utf-8&amp;oe=utf-8&amp;gws_rd=cr&amp;ei=ezZ0V-7iOoab6ASvlJe4Dg
{'ie': ['utf-8'], 'oe': ['utf-8'], 'gws_rd': ['cr'], 'ei': ['ezZ0V-7iOoab6ASvlJe4Dg']}
Ð¼Ð°ÑÐºÐ°Ð¸ Ð³ÐµÐ¹Ð»Ð°&amp;lr=10750&amp;clid=1985551-210&amp;win=213
{'win': ['213'], 'clid': ['1985551-210'], 'lr': ['10750']}
1&amp;q=ÐºÐ°Ðº+Ð²ÑÐ±ÑÐ°ÑÑ+ÑÐ¼Ð°ÑÑÑÐ¾Ð½
{'q': ['\xd0\xba\xd0\xb0\xd0\xba \xd0\xb2\xd1\x8b\xd0\xb1\xd1\x80\xd0\xb0\xd1\x82\xd1\x8c \xd1\x81\xd0\xbc\xd0\xb0\xd1\x80\xd1\x82\xd1\x84\xd0\xbe\xd0\xbd']}
Jade+Jantzen&amp;ie=utf-8&amp;oe=utf-8&amp;gws_rd=cr&amp;ei=FQB0V9WbIoahsAH5zZGACg
{'ie': ['utf-8'], 'oe': ['utf-8'], 'gws_rd': ['cr'], 'ei': ['FQB0V9WbIoahsAH5zZGACg']}
</code></pre>

<p>Is any way to get only text to all strings? </p>

<p>You can access the text using a dictionary lookup to get the list, then access the first element of the list:</p>

<pre><code>d = {'text': ['\xd0\xba\xd0\xbe\xd1\x80\xd0\xbe\xd0\xbb\xd0\xb5\xd0\xb2\xd1\x8b \xd0\xba\xd1\x80\xd0\xb8\xd0\xba\xd0\xb0 \xd1\x81\xd0\xbc\xd0\xbe\xd1\x82\xd1\x80\xd0\xb5\xd1\x82\xd1\x8c \xd0\xbe\xd0\xbd\xd0\xbb\xd0\xb0\xd0\xb9\xd0\xbd'], 'clid': ['2196598'], 'lr': ['213'], 'redircnt': ['1467230336.1']}
text = d['text'][0]

&gt;&gt;&gt; print text
ÐºÐ¾ÑÐ¾Ð»ÐµÐ²Ñ ÐºÑÐ¸ÐºÐ° ÑÐ¼Ð¾ÑÑÐµÑÑ Ð¾Ð½Ð»Ð°Ð¹Ð½
</code></pre>

<p>Or you can get it directly from the <code>parse_qs</code> result:</p>

<pre><code>&gt;&gt;&gt; print urlparse.parse_qs(s)['text'][0]
ÐºÐ¾ÑÐ¾Ð»ÐµÐ²Ñ ÐºÑÐ¸ÐºÐ° ÑÐ¼Ð¾ÑÑÐµÑÑ Ð¾Ð½Ð»Ð°Ð¹Ð½
</code></pre>

<p>To apply that to your code such that it will work for all values:</p>

<pre><code>print {k: v[0] for k, v in parse_qs(str).items()}
</code></pre>

<p>i.e. take the first item of each values list.</p>

<hr>

<p>If you want to print the dictionaries and have the strings appear in the proper representation, i.e. not as produced by <em>repr</em>, you could use the <code>json</code> module to dump the dictionary objects as strings, then print them:</p>

<pre><code>import json

d = {'text': ['\xd0\xba\xd0\xbe\xd1\x80\xd0\xbe\xd0\xbb\xd0\xb5\xd0\xb2\xd1\x8b \xd0\xba\xd1\x80\xd0\xb8\xd0\xba\xd0\xb0 \xd1\x81\xd0\xbc\xd0\xbe\xd1\x82\xd1\x80\xd0\xb5\xd1\x82\xd1\x8c \xd0\xbe\xd0\xbd\xd0\xbb\xd0\xb0\xd0\xb9\xd0\xbd'], 'clid': ['2196598'], 'lr': ['213'], 'redircnt': ['1467230336.1']}
s = json.dumps(d, ensure_ascii=False)

&gt;&gt;&gt; print s
{"text": ["ÐºÐ¾ÑÐ¾Ð»ÐµÐ²Ñ ÐºÑÐ¸ÐºÐ° ÑÐ¼Ð¾ÑÑÐµÑÑ Ð¾Ð½Ð»Ð°Ð¹Ð½"], "clid": ["2196598"], "lr": ["213"], "redircnt": ["1467230336.1"]}
</code></pre>

python urllib2 urllib urlparse