Python - multithreading with sockets gives random results
<p>I am really confused about my problem right now. I want to discover an open port over a list of hosts (or subnet).</p>

<p>So first let's show what I've done so far..    </p>

<pre><code>from multiprocessing.dummy import Pool as ThreadPool 
from netaddr import IPNetwork as getAddrList
import socket, sys

this = sys.modules[__name__]

def threading(ip):
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    sock.settimeout(this.timeout)

    failCode = 0

    try:
        if sock.connect_ex((str(ip), this.port)) == 0:
            #port open
            this.count += 1
        else:
            #port closed/filtered
            failCode = 1
            pass
    except Exception: 
        #host unreachable
        failCode = 2
        pass
    finally:
        sock.close()

#set thread num
threads = 64
#set socket timeout
this.timeout = 1
#set ip list
ipList = getAddrList('8.8.8.0/24')
#set port
this.port = 53
#set count
this.count = 0

#threading
Pool = ThreadPool(threads) 
Pool.map_async(threading, ipList).get(9999999)
Pool.close()
Pool.join()

#result
print str(this.count)
</code></pre>

<p>The Script works fine without any error. But I'm struggling about what it prints out..</p>

<p>So if I want to scan for example the subnet <code>8.8.8.0/24</code> and discover port <code>53</code>. I know the only server that has an open dns port is <code>8.8.8.8</code> (google-dns).</p>

<p>But when I run my script serveral times the <code>print str(this.count)</code> will randomly (as it seems to me..) return <code>0</code> or <code>1</code>.</p>

<p>What I also know:</p>

<ul>
<li>Scan only <code>8.8.8.8</code> prints always <code>1</code></li>
<li>Use only <em>1</em> thread for <code>/24</code> prints always <code>1</code></li>
<li>Use <em>256</em> threads for <code>/24</code> prints randomly <code>0</code> and <code>1</code></li>
<li>changing the timeout doesn't help</li>
</ul>

<p>So it seems like it has to do with the threading option that causes lags on my computer. But note that my CPU usage is &lt;10% and the network usage is &lt;50%! </p>

<p>But there is also another thing I don't understand..
If <code>print str(this.count)</code> returns <code>0</code> I would normally think it is because the threads are in conflict with each other and the socket doesn't get a connection.. but that isn't true because if <code>this.count</code> equals <code>0</code>, the <code>failCode</code> is set to <code>1</code> (on <code>8.8.8.8</code>)! Which means the port is closed.. but that must be also a bug of my script. I cannot think that this is caused by a lag of the server.. it's google there are no lags..</p>

<p>So additionally we know:</p>

<ul>
<li>output of <code>0</code> is because the server respond that the port is closed</li>
<li>and we are sure the port is definitely open </li>
</ul>

<p>So I also think about that if I have many threads that run <code>if sock.connect_ex((str(ip), this.port)) == 0:</code> at the same time maybe the host <code>8.8.8.8</code> looks in the wrong answer value. Maybe it struggles and look in the response of <code>8.8.8.7</code>. ..hope you know what I mean.  </p>

<p>**max_socket_connections is set to 512 on my linux distribution</p>

<p>Anybody experienced with socket threading and can explain me the situation or give me an answer how to prevent this? </p>

<p><strong>Solved:</strong></p>

<p>.. look at the answer ..</p>

<p><strong>The first mistake was:</strong></p>

<blockquote>
  <p>But note that my CPU usage is &lt;10% and the
  network usage is &lt;50%!</p>
</blockquote>

<p>Actually it was a 400 % network usage. I expected bits instead of bytes.<br>
Very embrassing..</p>

<p><strong>And the second mistake was:</strong></p>

<blockquote>
  <p>and we are sure the port is definitely open</p>
</blockquote>

<p>Google does actually block the port after ~5 attempts in a short period of time. They release the restriction after a few seconds or minutes.   </p>

python multithreading sockets connection multiprocessing