How to have one script call and run another one concurrently in python?
<p>What I am trying to accomplish is to stream tweets from Twitter for an hour, write the list of tweets to a file, clean and run analysis on the most recent hour of tweets, and then repeat the process indefinitely. </p>

<p>The problem I am running into is that if I run the cleaning and analysis of the tweets in the same script that's handling the streaming - by either hard-coding it or importing the functionality from a module - the whole script waits until these procedures are complete, and then begins again with the streaming. Is there a way to call the cleaning and analysis module within the streaming script so they run concurrently and the streaming doesn't stop while the cleaning and analysis is happening?</p>

<p>I've tried to achieve this by using <code>subprocess.call('python cleaner.py', shell=True)</code> and <code>subprocess.Popen('python cleaner.py', shell=True)</code>, but I don't really know how to use these tools properly, and the two examples above have resulted in the streaming being stopped, <code>cleaner.py</code> being run, and then the streaming resumed.</p>

<h2>Subprocess</h2>

<p>You can use <code>subprocess.Popen</code>, as you tried, to run a different script concurrently:</p>

<pre><code>the_other_process = subprocess.Popen(['python', 'cleaner.py'])
</code></pre>

<p>That line alone does what you want. What you <strong>don't want to</strong> do is:</p>

<pre><code>the_other_process.communicate()
# or
the_other_process.wait()
</code></pre>

<p>Those would stop current process and wait for the other one to finish. A very useful feature in other circumstances.</p>

<p>If you want to know whether the subprocess is finished (but not wait for it):</p>

<pre><code>result = the_other_process.poll()
if result is not None:
    print('the other process has finished and retuned %s' % result)
</code></pre>

<h2>Thread</h2>

<p>Concurrency can also be achieved using threads. In that case, you are not running a new process, you are just splitting the current process into concurrent parts. Try this:</p>

<pre><code>def function_to_be_executed_concurrently():
    for i in range(5):
        time.sleep(1)
        print('running in separate thread', i)

thread = threading.Thread(target=function_to_be_executed_concurrently)
thread.start()

for i in range(5):
    time.sleep(1)
    print('running in main thread', i)
</code></pre>

<p>The above code should result with mixed outputs of <code>running in separate thread</code> and <code>running in main thread</code>.</p>

<h2>Thread vs process</h2>

<ul>
<li>Using <code>subprocess</code>, you can run anything which could be run standalone from the shell. It does not have to be python.</li>
<li>Using <code>threading</code>, you can run any function in a concurrent thread of execution.</li>
<li>Threads share the same memory, so it is easy to share data between them (although there are issues when synchronization is needed). With processes, sharing data can become a problem. If a lot of data has to be shared, susbprocesses can be much slower.</li>
<li>Starting a new process is slower and consumes more resources than running a thread</li>
<li>Since threads run in the same process, they share are bound to the same <a href="https://wiki.python.org/moin/GlobalInterpreterLock" rel="nofollow">GIL</a>, which means most things will run on the same CPU core. If very slow CPU-consuming tasks need to be sped up, running them in separate processes my be faster.</li>
</ul>

<h2>Multiprocessing</h2>

<p><code>multiprocessing</code> module provides an interface similar to <code>threading</code>, but it runs subprocesses instead. This is useful when you need to take full advantage of all CPU cores.</p>

<hr>

<p>** Note that <code>subprocess.Popen(['python', 'cleaner.py'])</code> is the same thing as <code>subprocess.Popen('python cleaner.py', shell=True)</code>, but the former is better practice to learn.</p>

<p>For example, if there is a space in the path, this will fail:</p>

<pre><code>subprocess.Popen('python My Documents\\cleaner.py', shell=True)
</code></pre>

<p>It fails because it interprets <code>My</code> and <code>Documents\cleaner.py</code> as two separate arguments.</p>

<p>On the other hand, this will work as expected:</p>

<pre><code>subprocess.Popen(['python', 'My Documents\\cleaner.py'])
</code></pre>

<p>It works, because the arguments are explicitly separated by using a list.</p>

<p>The latter is especially superior if one of the arguments is in a variable:</p>

<pre><code>subprocess.Popen(['python', path_to_file])
</code></pre>

python concurrency subprocess