How to use standard Linux tools to fix a deadlocked script?
<p>I have a script in Python3 and if I use <code>subprocess.Popen.wait()</code> I have problem â my script iterates some Linux command many times and it looks to me like my app is not responding. When I use <code>subprocess.Popen.communicate()</code> my application correctly completes its work in a second.
<strong>What is the right way to solve this problem using Linux?</strong></p>

<p><strong>I think the solution must be somewhere in manipulating with buffer's variable</strong>, but I searched through the entire Internet and could not find anything suitable. May be I don't know enough structure and operation of Linux as a whole.</p>

<p>My question can be reformulated as follows: <strong>What's happened exactly when I use .wait() method? And that leads to failure of it? What is the cause of the so long waiting?</strong> When I aborting running task I see the next log: </p>

<pre><code>Traceback (most recent call last):
  File "./test.py", line 6, in &lt;module&gt;
    proc.wait()
  File "/usr/lib/python3.5/subprocess.py", line 1658, in wait
    (pid, sts) = self._try_wait(0)
  File "/usr/lib/python3.5/subprocess.py", line 1608, in _try_wait
    (pid, sts) = os.waitpid(self.pid, wait_flags)
KeyboardInterrupt
</code></pre>

<p>My files looks approximately like the next things:
script.py:</p>

<pre><code>#!/usr/bin/python3
# -*-coding: utf-8 -*-

import subprocess
proc = subprocess.Popen(['./1.py', '1000000'], stdin=subprocess.PIPE, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
proc.wait()
out = proc.stdout.read()
# out = proc.communicate()[0]
print(len(out))
</code></pre>

<p>1.py:</p>

<pre><code>#!/usr/bin/python3
# -*-coding: utf-8 -*-

import sys

x = sys.argv[-1]

# print(x, type(x))
for i in range(int(x)):
  print(i)
</code></pre>

<p>UPD: As we understand, the problem is a buffer overflow. It turns out the last version of question is, <strong>how to use the Linux possibility to expand buffer or redirect buffer to a file before running the script?</strong></p>

<p>UPD2: I also tried run the script as: $ <strong>python3 -u ./script.py</strong>, but, unfortunally, unbufferring doesn't work as I would like and script is hangs.</p>

<p>Your script is sending output to its <code>stdout</code> or <code>stderr</code> pipes. The operating system will buffer some data then block the process forever when the pipe fills. Suppose I have a long winded command like</p>

<p>longwinded.py:</p>

<pre><code>for i in range(100000):
    print('a'*1000)
</code></pre>

<p>The following hangs because the stdout pipe fills</p>

<pre><code>import sys
import subprocess as subp
p = subp.Popen([sys.executable, 'longwinded.py'], stdout=subp.PIPE,
    stderr=subp.PIPE)
p.wait()
</code></pre>

<p>The next one doesn't hang because <code>communicate</code> reads the stdout and stderr pipes into memory</p>

<pre><code>p = subp.Popen([sys.executable, 'longwinded.py'], stdout=subp.PIPE,
    stderr=subp.PIPE)
p.communicate()
</code></pre>

<p>If you don't care what stdout and err are, you can redirect them to the null device</p>

<pre><code>p = subp.Popen([sys.executable, 'longwinded.py'], 
    stdout=open(os.devnull, 'w'), 
    stderr=open(os.devnull, 'w'))
p.wait()
</code></pre>

<p>or save them to a file</p>

<pre><code>p = subp.Popen([sys.executable, 'longwinded.py'], 
    stdout=open('mystdout', 'w'), 
    stderr=open('mystderr', 'w'))
p.wait()
</code></pre>

python linux subprocess buffer buffer-overflow