how to search for elements containing unicode/arabic letters?
<p>I am running the below code to find an element containing Unicode Arabic characters. The below code works just fine if I replace XXX with English letter, however, if I replace them with Arabic letters It won't. </p>

<p>I checked the html page and it has "&lt; meta charset="utf-8" >" so I set the character set in my Py script at the first line just to make sure the letters are interpreted as expected but still not working.</p>

<p>Any clue is much appreciate it.</p>

<p>Thanks</p>

<pre><code># coding=UTF8

from selenium import webdriver
# create a new Firefox session
driver = webdriver.Firefox()
driver.implicitly_wait(10)
driver.get("http://www.norikoptic.com/Product/Women")
print driver.find_element_by_xpath(u"//*[contains(text(), 'XXX')]").text
</code></pre>

<p>I think you are not using the correct unicode in the xpath, 
check the demo in <code>Ipython</code> here</p>

<p>First I have selected one node to get the corresponding unicode for that arabic word, so after using that unicode modified the xpath as follows and this was the output.</p>

<pre><code>In [1]: response.xpath('//li[@class="lensItem"]/a/text()').extract()
Out[1]: [u'\u0639\u062f\u0633\u06cc']

In [2]: response.xpath(u'//a[contains(text(), "\u0639\u062f\u0633\u06cc")]/text()').extract()
Out[2]:
[u'\u0639\u062f\u0633\u06cc',
 u'\u0639\u062f\u0633\u06cc',
 u'\u0645\u0634\u062e\u0635\u0627\u062a \u0639\u062f\u0633\u06cc \u0622\u0641\u062a\u0627\u0628\u06cc']

In [3]: a = response.xpath(u'//a[contains(text(), "\u0639\u062f\u0633\u06cc")]/text()').extract()

In [4]: for i in a:
    ...:     print i
    ...:
Ø¹Ø¯Ø³Û
Ø¹Ø¯Ø³Û
ÙØ´Ø®ØµØ§Øª Ø¹Ø¯Ø³Û Ø¢ÙØªØ§Ø¨Û
</code></pre>

<p><strong>Edit</strong></p>

<p>I have tested the xpath using <code>Scrapy</code> but this will also work with <code>selenium</code>,</p>

<pre><code>In [6]: driver.find_element_by_xpath(u'//a[contains(text(), "\u0639\u062f\u0633\u06cc")]').text
Out[6]: u'\u0639\u062f\u0633\u06cc'
</code></pre>

<p>I hope this will help you to solve your issues.</p>

python selenium xpath unicode