Apache Lucene: How to use TokenStream to manually accept or reject a token when indexing
<p>I am looking for a way to write a custom index with Apache Lucene (PyLucene to be precise, but a Java answer is fine).</p>

<p>What I would like to do is the following : When adding a document to the index, Lucene will tokenize it, remove stop words, etc. This is usually done with the <code>Analyzer</code> if I am not mistaken.</p>

<p>What I would like to implement is the following : Before Lucene stores a given term, I would like to perform a lookup (say, in a dictionary) to check whether to keep the term or discard it (if the term is present in my dictionary, I keep it, otherwise I discard it). </p>

<p>How should I proceed ? </p>

<p>Here is (in Python) my custom implementation of the <code>Analyzer</code> :</p>

<pre><code>class CustomAnalyzer(PythonAnalyzer):

    def createComponents(self, fieldName, reader):

        source = StandardTokenizer(Version.LUCENE_4_10_1, reader)
        filter = StandardFilter(Version.LUCENE_4_10_1, source)
        filter = LowerCaseFilter(Version.LUCENE_4_10_1, filter)
        filter = StopFilter(Version.LUCENE_4_10_1, filter,
                            StopAnalyzer.ENGLISH_STOP_WORDS_SET)

        ts = tokenStream.getTokenStream()
        token = ts.addAttribute(CharTermAttribute.class_)
        offset = ts.addAttribute(OffsetAttribute.class_)

        ts.reset()

         while ts.incrementToken():
           startOffset = offset.startOffset()
           endOffset = offset.endOffset()
           term = token.toString()
           # accept or reject term 

         ts.end()
         ts.close()

           # How to store the terms in the index now ?

         return ????
</code></pre>

<p>Thank you for your guidance in advance !</p>

<p><strong>EDIT 1</strong> : After digging into Lucene's documentation, I figured it had something to do with the <code>TokenStreamComponents</code>. It returns a TokenStream with which you can iterate through the Token list of the field you are indexing.</p>

<p>Now there is something to do with the <code>Attributes</code> that I do not understand. Or more precisely, I can read the tokens, but have no idea how should I proceed afterward.</p>

<p><strong>EDIT 2</strong> : I found this <a href="http://stackoverflow.com/questions/2638200/how-to-get-a-token-from-a-lucene-tokenstream">post</a> where they mention the use of <code>CharTermAttribute</code>. However (in Python though) I cannot access or get a <code>CharTermAttribute</code>. Any thoughts ?</p>

<p><strong>EDIT3</strong> : I can now access each term, see update code snippet. Now what is left to be done is actually <em>storing the desired terms</em>...</p>

<p>The way I was trying to solve the problem was wrong. This <a href="http://stackoverflow.com/questions/24145688/how-to-tokenize-only-certain-words-in-lucene/24152554?noredirect=1#comment66511266_24152554">post</a> and <em>femtoRgon</em>'s answer were the solution.</p>

<p>By defining a filter extending <code>PythonFilteringTokenFilter</code>, I can make use of the function <code>accept()</code> (as the one used in the <code>StopFilter</code> for instance).</p>

<p>Here is the corresponding code snippet :</p>

<pre><code>class MyFilter(PythonFilteringTokenFilter):

  def __init__(self, version, tokenStream):
    super(MyFilter, self).__init__(version, tokenStream)
    self.termAtt = self.addAttribute(CharTermAttribute.class_)


  def accept(self):
    term = self.termAtt.toString()
    accepted = False
    # Do whatever is needed with the term
    # accepted = ... (True/False)
    return accepted
</code></pre>

<p>Then just append the filter to the other filters (as in the code snipped of the question) :</p>

<pre><code>filter = MyFilter(Version.LUCENE_4_10_1, filter)
</code></pre>

java python apache indexing lucene