Python - How to remotely execute processes in parallel and retrieve their output
<p>I'm trying to remotely execute a command on an unknown number of hosts (could be anywhere from one host to hundreds) in a Python script. The simple way of doing this is the following, but obviously it can get ridiculously time-consuming with many hosts:</p>

<pre><code>listOfOutputs = []
for host in listOfHosts:
  output = subprocess.Popen(shlex.split("ssh %s '&lt;command&gt;'" % host), stdout = subprocess.PIPE).communicate()[0]
  listOfOutputs.append(output)
</code></pre>

<p>Is there a way to do this same thing, but have the commands remotely execute in parallel so it doesn't take as long?</p>

<p>You have to run your <code>Popen.subprocess</code> calls each in a separate thread so you can launch as many as you want without blocking your main program.</p>

<p>I made a small example creating as many threads as there will be hosts. No big deal since threads will mostly wait for host reply (else, thread pools would have been better)</p>

<p>In my example, I have 3 hosts, and I perform a <code>ping</code> on each one. Outputs are stored in a thread-safe list of outputs and printed in the end:</p>

<pre><code>import threading
import subprocess

listOfOutputs=[]

lock = threading.Lock()

def run_command(args):
    p = subprocess.Popen(["ping","-n","1",args],stdout = subprocess.PIPE)
    output,err = p.communicate()
    lock.acquire()  # listOfOutputs is thread-safe now
    listOfOutputs.append(args+": "+output)
    lock.release()

threads=[]
listOfHosts = ['host1','host2','host3']
for host in listOfHosts:
    t = threading.Thread(target=run_command,args=[host])
    t.start()          # start in background
    threads.append(t)  # store thread object for future join operation


[t.join() for t in threads] # wait for all threads to finish

# print results
for o in listOfOutputs:
    print(o)
    print("-"*50)
</code></pre>

python ssh subprocess popen