Building a Python script which runs Perl scripts. How to redirect stdout?
<p>I am writing a Python script that use some Perl scripts, but one of them uses stdout so I must use a redirection <code>&gt;</code> in bash to write this output to a file.</p>

<p>All input and output files are text files.</p>

<pre><code># -*- coding: utf-8 -*-
import subprocess 

filename = input("What the name of the input? ")

#STAGE 1----------------------------------------------------------------------
subprocess.Popen(["perl", "run_esearch.pl", filename , 'result'])

#STAGE 2----------------------------------------------------------------------
subprocess.Popen(["perl", "shrink.pl", 'result']) 
'''Here the input from stage one is "shrunk" to smaller file, but
the output is printed to the console. Is it possible to write this out
to a file in Python, so I can use it in stage 3? '''

#STAGE 3----------------------------------------------------------------------
subprocess.Popen(["perl", "shrink2.pl", 'stdoutfromstage2'])
</code></pre>

<p>As far as I can tell, you have three Perl programs</p>

<ul>
<li><p><code>run_esearch.pl</code>, which expects two command-line parameters: the name of the input file and the name if the output file</p></li>
<li><p><code>shrink.pl</code>, which expects a single command-line parameter: the name of the input file. It writes its output to <code>stdout</code></p></li>
<li><p><code>shrink2.pl</code>, which expects a single command-line parameter: the name of the input file. You don't say anything about its output</p></li>
</ul>

<p>The standard, and most flexible way to write Linux programs is to have them read from <code>stdin</code> and write to <code>stdout</code>. That way input and output files may be specified explicitly on the command line using <code>&lt;</code> and <code>&gt;</code> redirection, or the same program may be used to read and write to a pipe <code>|</code> as part of a chain. Perl programs have the best of both worlds. Using an empty <code>&lt;&gt;</code> to read  the input will collect all data from files mentioned as command-line parameters, or will read from <code>stdin</code> if there are no parameters</p>

<p>I have no way of knowing which way your <em>shrink</em> programs treat their input, so I have to imagine the worst: that they explicitly open and read the file specified by the first parameter on the command line</p>

<p>Python's <code>subprocess</code> module provides the <code>Popen</code> constructor as well as several convenience functions. There is generally no need to use the constructor, especially if you are defaulting most of the parameters and discarding the returned object as you are</p>

<p>Since you're treating Python as a very high-level shell, I suggest that you pass shell command strings to <code>subprocess.call</code> with the <code>shell</code> parameter set to <code>True</code>. That will allow you to provide bash command strings, and you will be on more familiar ground and so feel more in control</p>

<pre class="lang-python prettyprint-override"><code>import subprocess 

filename = input("What's the name of the input? ")

subprocess.call("perl run_esearch.pl %s result" % filename, shell=True)

subprocess.call("perl shrink.pl result &gt; shrink1_out", shell=True)

subprocess.call("perl shrink2.pl shrink1_out", shell=True)
</code></pre>

<p>Note that this method is too risky to use in production code, as the response to <code>What the name of the input?</code> could contain malicious shell code that may compromise your system. But if the people using your program could just as easily destroy your system directly if they chose, then nothing is lost</p>

<p>Another issue is the use of fixed names for the intermediate files.
There is no guarantee that a separate independent process won't use a file with the same path, so in theory the process is insecure.
I followed your lead and used <code>result</code> for the output of <code>run_esearch.pl</code>, and invented <code>shrink1_out</code> for the output of <code>shrink.pl</code>, but a proper program would use the <code>tempfile</code> module and call <code>tempfile.NamedTemporaryFile</code> to create intermediate files that were guaranteed to be unique</p>

python bash perl subprocess stdout