Incomplete reads from file written by Popen()-based subprocess
<p>i'm trying to read the output from one function into another one.</p>

<p>if i break things down into two steps, call the first function(journal.py) from the command line, and then call the second(ip_list.py) i get the results that i'm looking for.</p>

<p>if i try to import the first and run it in the second the resulting list is empty.</p>

<pre><code>import re
import journal

journal.journal()
ip_list = []
with open('Invalid_names_file') as file1:
    print(file1)
    a = [re.search(r'((\d+\.)+\d+)', line).group() for line in file1]
    print(a)
    for x in a:
        if x not in ip_list:
            ip_list.append(x)

print(ip_list)
</code></pre>

<p>output -></p>

<pre><code>&lt;_io.TextIOWrapper name='Invalid_names_file' mode='r' encoding='UTF-8'&gt;
[]
[]
</code></pre>

<p>when called in this manner, the file that i'm opening is there (once the script is done running), with what i'm expecting, yet i can't get it to read from it when i try to include it with the import.</p>

<p>i have the print()s so that i can try to understand what's happening, yet can't get my head around it.</p>

<p>journal.py is a Popen command that writes the file.</p>

<p>edit for chris</p>

<p>journal.py</p>

<pre><code>from subprocess import Popen
import os

def journal():
    with open('Invalid_names_file', 'w') as Invalid_names_file:
        Popen('journalctl -u sshd.service --no-pager --since -168hours\
                --until today | grep Invalid', stdout=Invalid_names_file,\
                universal_newlines=True, bufsize=1, shell=True)
        if os.stat('Invalid_names_file').st_size == 0:
            Popen('journalctl -u ssh.service --no-pager --since -168hours\
                    --until today | grep Invalid', stdout=Invalid_names_file,\
                    universal_newlines=True, bufsize=1, shell=True)
    Invalid_names_file.close()
</code></pre>

<p>You should wait for <code>Popen()</code> to finish.
Assign its return value to a variable and call <code>wait()</code> on it:</p>

<pre><code>p = Popen('journalctl ...')
p.wait()
</code></pre>

<p>When you run the journal script separately, the parent process will only return when all of its children have terminated.
However, <code>Popen()</code> doesn't wait â unless you tell it to.
Thus, in your case, the <code>journal()</code> function exits immediately after starting the subprocess, so by the time you're reading the target file, it is still empty or incomplete.</p>

python file-io popen