Is there a faster way to make GET requests in Go?
<p>Consider this program:</p>

<pre><code>package main

import (
    "net/http"
    "os"
)

var url = "https://upload.wikimedia.org/wikipedia/commons/f/fe/FlumeRide%2C_Liseberg_-_last_steep_POV.ogv"

func main() {
    response, _ := http.Get(url)
    defer response.Body.Close()

    f, _ := os.Create("output.ogv")
    defer f.Close()

    _, err = io.Copy(f, response.Body)
}
</code></pre>

<p>It has the same functionality as <code>wget $url</code> and takes ~<strong>7.3 seconds</strong> to run (for me). <code>wget</code> takes only ~<strong>4.6 seconds</strong>. Why the huge discrepancy? This trivial Python program, which loads the entire video into memory before writing it to disk, takes around <strong>5.2 seconds</strong>:</p>

<pre><code>import requests

url = "https://upload.wikimedia.org/wikipedia/commons/f/fe/FlumeRide%2C_Liseberg_-_last_steep_POV.ogv"

def main():
    r = requests.get(url)
    with open('output.ogv','wb') as output:
        output.write(r.content)

if __name__ == "__main__":
    main()
</code></pre>

<h1>Profiling</h1>

<p>I've investigated this quite a bit. Here are some approaches I've taken:</p>

<ol>
<li>Use different buffer sizes in <code>io.Copy</code></li>
<li>Use other Readers/Writers</li>
<li>Concurrency / parallelism</li>
<li>Downloading larger files</li>
</ol>

<h3>Different buffer sizes</h3>

<p>I tried many different buffer sizes using <code>io.CopyBuffer</code>, and I found that the default buffer size of 32KB leaves me with the best speeds (which are still 1.6 to 1.8 times slower than <code>wget</code> and Python's <code>reqeusts</code>).</p>

<h3>Other Readers/Writers</h3>

<p>All of the other readers and writers were negligibly slower than using <code>io.Copy</code>. I tried using <code>(f *File) Write</code> and some other buffered readers/writers.</p>

<h3>Concurrency / Parallelism</h3>

<p>I even wrote a fairly long program that uses <code>range</code> in headers to download this file in parallel, but as expected I didn't seem any remarkable improvements in speed.</p>

<h3>Larger files</h3>

<p>I downloaded a file more than three times as large as this one, and my Go implementation is still 1.5 to 2 times slower than wget and requests.</p>

<h1>Other Things of Note</h1>

<ol>
<li>I'm building a binary before timing anything.</li>
<li>The vast majority of the time is spent on actually writing/copying <code>response.Body</code>. The part seems to account for all but ~0.3 seconds of elapsed time, regardless of how large the file I'm downloading is.</li>
</ol>

<hr>

<p>So what am I doing wrong? Should I just expect GET requests to take this much longer in Go?</p>

<p>I'm not sure what to tell you.  I just tried to duplicate your findings, but for me, all 3 versions take roughly the same amount of time</p>

<pre><code>wget   8.035s  
go     8.174s
python 8.242s
</code></pre>

<p>Maybe try the same experiment inside a clean VM or docker container?</p>

python http go wget