Copying a file to multiple paths at the same time
<p>I am transferring a 150-200mb file to many locations (shared drives located across the world) daily. The issue is that each transfer (using shutil) takes probably 100-700 seconds and each one has to complete in order for the next one to begin. It now takes like a full hour to transfer some files if I do it that way. My temporary solution was to create a separate .py file to run for each location so they can be done simultaneously, but that is not ideal.</p>

<p>How can I dip into multi-thread programming? I'd like to run all of the transfers at once but I have zero experience with this.</p>

<p>A simple google search landed me with:</p>

<p><a href="https://docs.python.org/3/library/concurrent.futures.html" rel="nofollow">https://docs.python.org/3/library/concurrent.futures.html</a>.</p>

<pre><code>import shutil
with ThreadPoolExecutor(max_workers=4) as e:
    e.submit(shutil.copy, 'src1.txt', 'dest1.txt')
    e.submit(shutil.copy, 'src2.txt', 'dest2.txt')
    e.submit(shutil.copy, 'src3.txt', 'dest3.txt')
    e.submit(shutil.copy, 'src4.txt', 'dest4.txt')
</code></pre>

<p>Can someone point me into the right direction? I have been meaning to learn how to do things in parallel for a while now but never got around to it.</p>

<p>Here's a working example that does what you want. Note that it may not be any faster than one-at-a-time if the bottleneck is network bandwidth.</p>

<pre><code>from concurrent.futures import ThreadPoolExecutor
import os
import shutil
import time
from threading import Lock

src_dir = './test_src'
src_files = 'src1.txt', 'src2.txt', 'src3.txt', 'src4.txt'
dst_dir = './test_dst'
print_lock = Lock()

_print = print  # save original
def print(*args, **kwargs):
    """Prevents concurrent printing."""
    with print_lock:
        _print(*args, **kwargs)

def copy_file(src_file):
    src_file = os.path.join(src_dir, src_file)
    print('starting transfer of "{}"'.format(src_file))
    shutil.copy2(src_file, dst_dir)
    print('transfer of "{}" completed'.format(src_file))

with ThreadPoolExecutor(max_workers=4) as e:
    jobs = [e.submit(copy_file, src_file) for src_file in src_files]

while any(job.running() for job in jobs):
    time.sleep(.1)
print('done')
</code></pre>

python python-3.x shutil