How to remove defined text (http headers) from a text file
<p>I have a data set of documents containing http headers. I want to go through these documents remove these headers while leaving rest of the text. How can I do that?</p>

<pre><code>WARC/1.0
WARC-Type: response
WARC-Date: 2012-02-10T21:58:44Z
WARC-TREC-ID: clueweb12-0000wb-76-38422
WARC-IP-Address: 207.241.148.80
WARC-Payload-Digest: sha1:W6JMWCNM43FDYNW466OADMH2KDGKJCGR
WARC-Target-URI: http://someurl.http
WARC-Record-ID: &lt;urn:uuid:5a783f09-f0d8-4564-8f3a-c0d1ace7177b&gt;
Content-Type: application/http; msgtype=response
Content-Length: 26043

HTTP/1.1 200 OK
Date: Fri, 10 Feb 2012 21:58:45 GMT
Server: Apache
Vary: *
PRAGMA: no-cache
P3P: CP="IDC DSP COR DEVa TAIa OUR BUS UNI"
Cache-Control: max-age=-3600
Expires: Fri, 10 Feb 2012 20:58:45 GMT
Connection: close
Content-Type: text/html
</code></pre>

<p>This will do what you say you want.
It will leave the original file alone and put the cleaned version into a new file.</p>

<pre><code>datafile = 'test1.txt'
outputfile = 'output.txt'

with open(outputfile, encoding='utf-8', mode='w') as outfile:
    with open(datafile, encoding='utf-8', mode='r') as infile:
        foundhdrstart = False

        for line in infile:
            if line.strip() == 'WARC/1.0':
                foundhdrstart = True
            if foundhdrstart is False:
                outfile.write(line)
            if line.strip() == 'Content-Type: text/html':
                foundhdrstart = False
</code></pre>

python