Paramiko Buffer issue
<p>I have a problem of buffer using paramiko, I found the same question <a href="http://stackoverflow.com/q/12486623/2662302">here</a> and one of the solutions states that:</p>

<blockquote>
  <p>Rather than using .get(), if you just call .open() to get an SFTPFile
  instance, then call .read() on that object, or just hand it to the
  Python standard library function shutil.copyfileobj() to download the
  contents. That should avoid the Paramiko prefetch cache, and allow you
  to download the file even if it's not quite as fast.</p>
</blockquote>

<p>Now if I have:</p>

<pre><code>ssh=paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(host,username=user,password=pwd)
sftp=ssh.open_sftp()
sftp_file=sftp.open(remote_file_adress)
</code></pre>

<p>How I save this file-like-object in a csv in my local pc? (the original file is a csv also)</p>

<p>Here is a working example that fetches copies a test file on your local machine. The file is much smaller than 1 GIG but gives the general plan.</p>

<pre><code>import paramiko
import os
import shutil
import time
import getpass

# get params
user = getpass.getuser()
pwd = getpass.getpass("Enter password: ")
bufsize = 2**20
host = 'localhost'
test_file_lines = 1000000

# create test file
now = time.asctime()
testfile_path = os.path.abspath('deleteme')
local_path = 'deleteme.copy'

print('writing test file...')
start = time.time()
with open(testfile_path, 'w') as fp:
    for _ in range(test_file_lines):
        fp.write(now + '\n')
delta = time.time() - start
file_size = os.stat(testfile_path).st_size
print("file size %d, %d KB/Sec" % (file_size, file_size/1024/delta))

# make connection
ssh=paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(host,username=user,password=pwd)
sftp=ssh.open_sftp()
sftp_file=sftp.open(testfile_path, bufsize=bufsize)
print('copying file...')
start = time.time()
shutil.copyfileobj(sftp_file, 
    open(local_path, 'wb', bufsize),
    bufsize)
delta = time.time() - start
print('%.3f seconds, %d KB/Sec' % (delta, file_size/1024/delta))

#assert open(testfile_path).read() == open(local_path).read(), "files match"
</code></pre>

<p>Running on my machine I got</p>

<pre><code>Enter password: 
writing test file...
file size 25000000, 21017 KB/Sec
copying file...
10.225 seconds, 2387 KB/Sec
</code></pre>

<p>We expect some slow down because there is a read and a write plus network costs (its to local host so doesn't really touch the wire), but that does seem kinda slow. I am using a low-powered laptop with 2 cores and between this app and sshd, used much of the cpu, presumably to do the encryption. A higher powered machine may work better.</p>

python csv paramiko