regular expression unicode character does not match
<p>I am trying to use regular expression over a text that contains some special character like Ã ,Ã¨,Ã¹ etc. </p>

<pre><code>filter_2 = ur'(?:^\|\s+)?(?:(?:main_interests)|(?:influenced)|(?:influences))\s+?=[\s\W]+?(?:[\w}])*?([\d\w\s\-()*â&amp;;\[\]|.&lt;&gt;:/",\']*)(?=\n)'
compiled = re.compile(filter_2, flags=re.U | re.M)
filter_list = re.findall(compiled, information)
</code></pre>

<p>The text below is the result of the evaluation of the expression.</p>

<blockquote>
  <p>[[Pedro CalderÃ³n de la Barca|CalderÃ³n]], [[Christian FÃ¼rchtegott Gellert|Gellert]], [[Oliver Goldsmith|Goldsmith]], [[Hafez]], [[Johann Gottfried Herder|Herder]], [[Homer]], [[KÄlidÄsa]], [[Kant]], [[Friedrich Gottlieb Klopstock|Klopstock]], [[Gotthold Ephraim Lessing|Lessing]], [[Carl Linnaeus|Linnaeus]], [[James Macpherson|Macpherson]], [[Jean-Jacques Rousseau|Rousseau]], [[Friedrich Schiller|Schiller]], [[William Shakespeare|Shakespeare]], [[Spinoza]], [[Emanuel Swedenborg|Swedenborg]],[[Karl Robert Mandelkow]], Bodo Morawe: Goethes Briefe. 2. edition. Vol. 1: Briefe der Jahre 1764â1786. ''Christian Wegner'', Hamburg 1968, p.&nbsp;709 [[Johann Joachim Winckelmann|Winckelmann]]`</p>
</blockquote>

<p>Now, when i try to use another regular expression over the above text in order to extrapolate the words in the square brackets, the result is wrong. All the words that represent a special character, like Ã  Ã¹ or Ã¨, are removed and the result is not the one expected.</p>

<pre><code>filter_6 = ur'(?&lt;=\[\[)([\w\s.-]+)((?=]])|(?=|))'
another_compiled = re.compile(filter_6, flags=re.U | re.M)
another_filtered_list = re.findall(another_compiled, (str(filter_list)))
</code></pre>

<p>These are my results:</p>

<blockquote>
  <p>[('Pedro Calder', ''), ('Christian F', ''), ('Oliver Goldsmith', ''), ('Hafez', ''), ('Johann Gottfried Herder', ''), ('Homer', ''), ('K', ''), ('Kant', ''), ('Friedrich Gottlieb Klopstock', ''), ('Gotthold Ephraim Lessing', ''), ('Carl Linnaeus', ''), ('James Macpherson', ''), ('Jean-Jacques Rousseau', ''), ('Friedrich Schiller', ''), ('William Shakespeare', ''), ('Spinoza', ''), ('Emanuel Swedenborg', ''), ('Karl Robert Mandelkow', ''), ('Johann Joachim Winckelmann', ''), ('Thomas Carlyle', ''), ('Ernst Cassirer', ''), ('Charles Darwin', ''), ('Sigmund Freud', ''), ('G', ''), ('Andr', ''), ('Hermann Hesse', ''), ('G.W.F. Hegel', ''), ('Muhammad Iqbal', ''), ('Daisaku Ikeda', ''), ('Carl Gustav Jung', ''), ('Milan Kundera', ''), ('S', ''), ('Jean-Baptiste Lamarck', ''), ('Joaquim Maria Machado de Assis', ''), ('Thomas Mann', ''), ('Friedrich Nietzsche', ''), ('France Pre', ''), ('Grigol Robakidze', ''), ('Friedrich Schiller', ''), ('Oswald Spengler', ''), ('Max Stirner', ''), ('Friedrich Wilhelm Joseph Schelling', ''), ('Arthur Schopenhauer', ''), ('Oswald Spengler', ''), ('Rudolf Steiner', ''), ('Henry David Thoreau', ''), ('Nikola Tesla', ''), ('Ivan Turgenev', ''), ('Ludwig Wittgenstein', ''), ('Richard Wagner', ''), ('Leopold von Ranke', '')]</p>
</blockquote>

<p>These are the results i would like to achieve</p>

<blockquote>
  <p>MATCH 1
  1.  [2-28]  <code>Pedro CalderÃ³n de la Barca</code>
  MATCH 2
  1.  [43-72] <code>Christian FÃ¼rchtegott Gellert</code>
  MATCH 3
  1.  [86-102]    <code>Oliver Goldsmith</code>
  MATCH 4
  1.  [118-123]   <code>Hafez</code>
  MATCH 5
  1.  [129-152]   <code>Johann Gottfried Herder</code>
  MATCH 6
  1.  [165-170]   <code>Homer</code>
  MATCH 7
  1.  [176-184]   <code>KÄlidÄsa</code>
  MATCH 8
  1.  [190-194]   <code>Kant</code>
  MATCH 9
  1.  [200-228]   <code>Friedrich Gottlieb Klopstock</code>
  MATCH 10
  1.  [244-268]   <code>Gotthold Ephraim Lessing</code>
  MATCH 11
  1.  [282-295]   <code>Carl Linnaeus</code>
  MATCH 12
  1.  [310-326]   <code>James Macpherson</code>
  MATCH 13
  1.  [343-364]   <code>Jean-Jacques Rousseau</code>
  MATCH 14
  1.  [379-397]   <code>Friedrich Schiller</code>
  MATCH 15
  1.  [412-431]   <code>William Shakespeare</code>
  MATCH 16
  1.  [449-456]   <code>Spinoza</code>
  MATCH 17
  1.  [462-480]   <code>Emanuel Swedenborg</code>
  MATCH 18
  1.  [501-522]   <code>Karl Robert Mandelkow</code>
  MATCH 19
  1.  [659-685]   <code>Johann Joachim Winckelmann</code></p>
</blockquote>

<p>All the regular expression are tested online and they work perfectly. There is a way to actually include these special characters?</p>

<p>In <strong>Python 3</strong>, the regex doesn't compile. This seemed to work for me when I changed:</p>

<pre><code>filter_6 = ur'(?&lt;=\[\[)([\w\s.-]+)((?=]])|(?=|))'
</code></pre>

<p>to just a unicode (not raw) string:</p>

<pre><code>filter_6 = u'(?&lt;=\[\[)([\w\s.-]+)((?=]])|(?=|))'
</code></pre>

<p>In <strong>Python 2</strong>, I believe the issue is the casting of the list to a string. Changing <code>str(filter_list)</code> to <code>' '.join(filter_list)</code> seemed to work for me.</p>

python regex unicode