iterating through passages for string extrction
<p>I came across this problem from a forum where these things are needed to be done:
You will be given a sequence of passages, and must filter out any passage whose text (sequence of whitespace-delimited words) is wholly contained as a sub-passage of one or more of the other passages.</p>

<p>When comparing for containment, certain rules must be followed:
The case of alphabetic characters should be ignored
Leading and trailing whitespace should be ignored
Any other block of contiguous whitespace should be treated as a single space
non-alphanumeric character should be ignored, white space should be retained
Duplicates must also be filtered - if two passages are considered equal with respect to the comparison rules listed above, only the shortest should be retained. If they are also the same length, the earlier one in the input sequence should be kept. The retained passages should be output in their original form (identical to the input passage), and in the same order.</p>

<p>Input1: IBM cognitive computing|IBM "cognitive" computing is a revolution| ibm cognitive computing|'IBM Cognitive Computing' is a revolution? </p>

<p>Output1: IBM "cognitive" computing is a revolution</p>

<p>Input2: IBM cognitive computing|IBM "cognitive" computing is a revolution|the cognitive computing is a revolution</p>

<p>Output2: IBM "cognitive" computing is a revolution|the cognitive computing is a revolution</p>

<p>I wrote the following code in python, but it's giving me some other output rather than the first test case:</p>

<pre><code>f = open("input.txt",'r')
s = (f.read()).split('|')
str = ''
for a in s:
    for b in s:
        if(''.join(e for e in a.lower() if e.isalnum()))not in (''.join(e for e in b.lower() if e.isalnum())):
            str = a.translate(None, "'?")

print str 
</code></pre>

<p><code>input.txt</code> contains the first test case input. And I am getting the output as :
<strong>IBM Cognitive Computing is a revolution</strong>. 
Could someone please chime in and help me. Thanks</p>

<p>I literally coded this for you, hopefully it's easy to understand (I made it less optimal as such). Let me know if you need to speed it up or if you have any questions!</p>

<p>BTW, this is python 3, just remove the brackets from the <code>print</code> statements and it will copy and paste and run for python 2.</p>

<pre><code>import re

def clean_input(text):
    #non-alphanumeric character should be ignored
    text = re.sub('[^a-zA-Z\s]', '', text)
    #Any other block of contiguous whitespace should be treated as a single space
    #white space should be retained
    text = re.sub(' +',' ',text)
    #Leading and trailing whitespace should be ignored
    text = text.strip(' \t\n\r')
    # You probably want this too
    text = text.lower()
    return text

def process(text):
    #If they are also the same length, the earlier one in the input sequence should be kept.
    # Using arrays (can use OrderedDict too, probably easier and nice, although below is clearer for you.
    original_parts = text.split('|')
    clean_parts = [clean_input(x) for x in original_parts]
    original_parts_to_check = []
    ignore_idx = []
    for idx, ele in enumerate(original_parts):
        if idx in ignore_idx:
            continue
        #The case of alphabetic characters should be ignored
        if len(ele) &lt; 2:
            continue
        #Duplicates must also be filtered -if two passages are considered equal with respect to the comparison rules listed above, only the shortest should be retained.
        if clean_parts[idx] in clean_parts[:idx]+clean_parts[idx+1:]:
            indices = [i for i, x in enumerate(clean_parts) if x == clean_parts[idx]]
            use = indices[0]
            for i in indices[1:]:
                if len(original_parts[i]) &lt; len(original_parts[use]):
                    use = i
            if idx == use:
                ignore_idx += indices
            else:
                ignore_idx += [x for x in indices if x != use]
                continue
        original_parts_to_check.append(idx)
    # Doing the text in text matching here. Depending on size and type of dataset,
    # Which you should test as it would affect this, you may want this as part
    # of the function above, or before, etc. If you're only doing 100 bits of
    # data, then it doesn't matter. Optimize accordingly.
    text_to_return = []
    clean_to_check = [clean_parts[x] for x in original_parts_to_check]
    for idx in original_parts_to_check:
        # This bit can be done better, but I have no more time to work on this.
        if any([(clean_parts[idx] in clean_text) for clean_text in [x for x in clean_to_check if x != clean_parts[idx]]]):
            continue
        text_to_return.append(original_parts[idx])
    #The retained passages should be output in their original form (identical to the input passage), and in the same order.
    return '|'.join(text_to_return)

assert(process('IBM cognitive computing|IBM "cognitive" computing is a revolution| ibm cognitive computing|\'IBM Cognitive Computing\' is a revolution?') ==
       'IBM "cognitive" computing is a revolution')
print(process('IBM cognitive computing|IBM "cognitive" computing is a revolution| ibm cognitive computing|\'IBM Cognitive Computing\' is a revolution?'))
assert(process('IBM cognitive computing|IBM "cognitive" computing is a revolution|the cognitive computing is a revolution') ==
       'IBM "cognitive" computing is a revolution|the cognitive computing is a revolution')
print(process('IBM cognitive computing|IBM "cognitive" computing is a revolution|the cognitive computing is a revolution'))
</code></pre>

<p>Also, if this helped you, would be great to get some points, so an accept would be nice :) (I see you're new here).</p>

python string