How do I delete duplicate lines and create a new file without duplicates?
<p>I searched on here an found many postings, however none that I can implement into the following code</p>

<pre><code>with open('TEST.txt') as f:
    seen = set()
    for line in f:
        line_lower = line.lower()
        if line_lower in seen and line_lower.strip():
            print(line.strip())
        else:
            seen.add(line_lower)
</code></pre>

<p>I can find the duplicate lines inside my TEST.txt file which contains hundreds of URLs.</p>

<p>However I need to remove these duplicates and create a new text file with these removed and all other URLs intact.</p>

<p>I will be Checking this newly created file for 404 errors using r.status_code.</p>

<p>In a nutshell I basically need help getting rid of duplicates so I can check for dead links. thanks for your help.</p>

<p>this is something you could use:</p>

<pre><code>import linecache

with open('pizza11.txt') as f:
    for i, l in enumerate(f):
            pass
    x=i+1
    k=0
    i=2
    j=1
    initial=linecache.getline('pizza11.txt', 1)
    clean= open ('clean.txt','a')
    clean.write(initial)
    while i&lt;(x+1):
        a=linecache.getline('pizza11.txt', i)
        while j&lt;i:
            b=linecache.getline('pizza11.txt', j)
            if a==b:
                k=k+1
            j=j+1
        if k==0:
                clean= open ('clean.txt','a')
                clean.write(a)
        k=0
        j=1
        i=i+1
</code></pre>

<p>With this you are going through every line and checking with the ones before itself, if there are no matches with the previous written lines then it adds it on the document.</p>

<p>pizza11 is the name of a file I have on my computer which is a text file with a ton of stuff in a list that I use to try stuff like this out, you would just need to change that to whatever your starting file is. Your output file with no duplicates would be clean.txt</p>

python hyperlink duplicates