Javascript regexp with unicode and punctuation
<p>I have the follow test case for splitting unicoded words but don't know how to do in it javascript.</p>

<pre><code>describe("garden: utils", () =&gt; {
  it("should split correctly", () =&gt; {
    assert.deepEqual(segmentation('Hockey is a popular sport in Canada.'), [
      'Hockey', 'is', 'a', 'popular', 'sport', 'in', 'Canada', '.'
    ]);

    assert.deepEqual(segmentation('How many provinces are there in Canada?'), [
      'How', 'many', 'provinces', 'are', 'there', 'in', 'Canada', '?'
    ]);

    assert.deepEqual(segmentation('The forest is on fire!'), [
      'The', 'forest', 'is', 'on', 'fire', '!'
    ]);

    assert.deepEqual(segmentation('Emily Carr, who was born in 1871, was a great painter.'), [
      'Emily', 'Carr', ',', 'who', 'was', 'born', 'in', '1871', ',', 'was', 'a', 'great', 'painter', '.'
    ]);

    assert.deepEqual(segmentation('This is David\'s computer.'), [
      'This', 'is', 'David', '\'', 's', 'computer', '.'
    ]);

    assert.deepEqual(segmentation('The prime minister said, "We will win the election."'), [
      'The', 'prime', 'minister', 'said', ',', '"', 'We', 'will', 'win', 'the', 'election', '.', '"'
    ]);

    assert.deepEqual(segmentation('There are three positions in hockey: goalie, defence, and forward.'), [
      'There', 'are', 'three', 'positions', 'in', 'hockey', ':', 'goalie', ',', 'defence', ',', 'and', 'forward', '.'
    ]);

    assert.deepEqual(segmentation('The festival is very popular; people from all over the world visit each year.'), [
      'The', 'festival', 'is', 'very', 'popular', ';', 'people', 'from', 'all', 'over', 'the', 'world',
      'visit', 'each', 'year', '.'
    ]);

    assert.deepEqual(segmentation('Mild, wet, and cloudy - these are the characteristics of weather in Vancouver.'), [
      'Mild', ',', 'wet', ',', 'and', 'cloudy', '-', 'these', 'are', 'the', 'characteristics', 'of', 'weather',
      'in', 'Vancouver', '.'
    ]);

    assert.deepEqual(segmentation('sweet-smelling'), [
      'sweet', '-', 'smelling'
    ]);
  });

  it("should not split unicoded words", () =&gt; {
    assert.deepEqual(segmentation('hacer a propÃ³sito'), [
      'hacer', 'a', 'propÃ³sito'
    ]);

    assert.deepEqual(segmentation('nhÃ  em cÃ³ con mÃ¨o'), [
      'nhÃ ', 'em', 'cÃ³', 'con', 'mÃ¨o'
    ]);
  });

  it("should group periods", () =&gt; {
    assert.deepEqual(segmentation('So are ... the fishes.'), [
      'So', 'are', '...', 'the', 'fishes', '.'
    ]);

    assert.deepEqual(segmentation('So are ...... the fishes.'), [
      'So', 'are', '......', 'the', 'fishes', '.'
    ]);

    assert.deepEqual(segmentation('arriba arriba ja....'), [
      'arriba', 'arriba', 'ja', '....'
    ]);
  });
});
</code></pre>

<p>Here is the equivalent expression in python:</p>

<pre><code>class Segmentation(BaseNLPProcessor):
    pattern = re.compile('((?u)\w+|\.{2,}|[%s])' % string.punctuation)

    @classmethod
    def ignore_value(cls, value):
        # type: (str) -&gt; bool
        return negate(compose(is_empty, string.strip))(value)

    def split(self):
        # type: () -&gt; List[str]
        return filter(self.ignore_value, self.pattern.split(self.value()))
</code></pre>

<p>I want to write a equivalent function in python for javascript to split by unicoded words and punctuation, group by multiple dots ...</p>

<pre><code>Segmentation("Hockey is a popular sport in Canada.").split()
</code></pre>

<p>Pretty complicated given there's no negative look-behind assertions in JavaScript RegExp, and Unicode support is not official yet (only supported in Firefox by a flag at the moment). This uses a library (XRegExp) to handle the unicode classes. If you need the full normal regular expression, <strong><em>it's huge</em></strong>. Just comment and let me know, and I'll update the answer to use the exploded normal RegExp statements that include the Unicode ranges.</p>

<pre><code>const rxLetterToOther = XRegExp('(\\p{L})((?!\\s)\\P{L})','g');
const rxOtherToLetter = XRegExp('((?!\\s)\\P{L})(\\p{L})','g');
const rxNumberToOther = XRegExp('(\\p{N})((?!\\s)\\P{N})','g');
const rxOtherToNumber = XRegExp('((?!\\s)\\P{N})(\\p{N})','g');
const rxPuctToPunct = XRegExp('(\\p{P})(\\p{P})','g');
const rxSep = XRegExp('\\s+','g');

function segmentation(s) {
  return s
    .replace(rxLetterToOther, '$1 $2')
    .replace(rxOtherToLetter, '$1 $2')
    .replace(rxNumberToOther, '$1 $2')
    .replace(rxOtherToNumber, '$1 $2')
    .replace(rxPuctToPunct, '$1 $2')
    .split(rxSep);
}
</code></pre>

<p><a href="https://jsfiddle.net/a3tf68ae/14/" rel="nofollow">Here it is passing all the test cases!</a></p>

<p><div class="snippet" data-lang="js" data-hide="false" data-console="true" data-babel="false">
<div class="snippet-code">
<pre class="snippet-code-js lang-js prettyprint-override"><code>window.onbeforeunload = "";</code></pre>
<pre class="snippet-code-css lang-css prettyprint-override"><code>* { margin: 0; padding: 0; border: 0; overflow: hidden; }
object { width: 100%; height: 100%; width: 100vw; height: 100vh; }</code></pre>
<pre class="snippet-code-html lang-html prettyprint-override"><code>&lt;object data="https://fiddle.jshell.net/a3tf68ae/14/show/" /&gt;</code></pre>
</div>
</div>
</p>

<p><em>Edit: Updated the test case to print the huge RegExp sources beneath the test results. Run the snippet to see the embedded test case.</em></p>

javascript python regex unicode punctuation