Searching for and manipulating the content of a keyword in a huge file
<p>I have a huge HTML file that I have converted to text file. (The file is Facebook home page's source). Assume the text file has a specific keyword in some places of it. For example: "some_keyword: [bla bla]". How would I print all the different bla blas that are followed by some_keyword?</p>

<pre><code>{id:"1126830890",name:"Hillary Clinton",firstName:"Hillary"}
</code></pre>

<p>Imagine there are 50 different names with this format in the page. How would I print all the names followed by "name:", considering the text is very large and crashes when you read() it or try to search through its lines. </p>

<p>Sample File: </p>

<p><code>shortProfiles:{"100000094503825":{id:"100000094503825",name:"Bla blah",firstName:"Blah",vanity:"blah",thumbSrc:"https://scontent-lax3-1.xx.fbcdn.net/v/t1.0-1/c19.0.64.64/p64x64/10354686_10150004552801856_220367501106153455_n.jpg?oh=3b26bb13129d4f9a482d9c4115b9eeb2&amp;oe=5883062B",uri:"https://www.facebook.com/blah",gender:2,i18nGender:16777216,type:"friend",is_friend:true,mThumbSrcSmall:null,mThumbSrcLarge:null,dir:null,searchTokens:["Bla"],alternateName:"",is_nonfriend_messenger_contact:false},"1347968857":</code></p>

<p>I just wanted to throw this out there even though I agree with all the comments about just dealing with the html directly or using Facebook's API (probably the safest way), but open file objects in Python can be used as a generator yielding lines without reading the entire file into memory and the re module can be used to extract information from text.</p>

<p>This can be done like so:</p>

<pre><code>import re

regex = re.compile(r"(?:some_keyword:\s\[)(.*?)\]")

with open("filename.txt", "r") as fp:
    for line in fp:
        for match in regex.findall(line):
            print(match)
</code></pre>

<p>Of course this only works if the file is in a "line-based" format, but the end effect is that only the line you are on is loaded into memory at any one time. </p>

<p><a href="https://docs.python.org/2/library/re.html" rel="nofollow">here</a> is the Python 2 docs for the <code>re</code> module </p>

<p><a href="https://docs.python.org/3.5/library/re.html" rel="nofollow">here</a> is the Python 3 docs for the <code>re</code> module</p>

<p>I cannot find documentation which details the generator capabilities of file objects in Python, it seems to be one of those well-known secrets...Please feel free to edit and remove this paragraph if you know where in the Python docs this is detailed. </p>

python