gRPC client side load balancing
<p>I'm using gRPC with Python as client/server inside kubernetes pods...
I would like to be able to launch multiple pods of the same type (gRPC servers) and let the client connect to them (randomly).</p>

<p>I dispatched 10 pods of the server and setup a 'service' to target them. Then, in the client, I connected to the DNS name of the service - meaning kubernetes should do the load-balancing and direct me to a random server pod.
In reality, the client calls the gRPC functions (which works well) but when I look at the logs I see that all calls going to the same server pod. </p>

<p>I presume the client is doing some kind of DNS caching which leads to all calls being sent to the same server. Is this the case? Is there anyway to disable it and set the same stub client to make a "new" call and fetch a new ip by DNS with each call? </p>

<p>I am aware of the overhead I might cause if it will query the DNS server each time but distributing the load is much more important for me at the moment.</p>

<p><strong>EDIT</strong></p>

<p>probably not a caching issue... Might just be the way gRPC works. 
HTTP/2 and persistent reusable connection. Any way to "disconnect" after each call?</p>

<p>If you've created a vanilla Kubernetes service, the service should have its own load-balanced virtual IP (check if <code>kubectl get svc your-service</code> shows a <code>CLUSTER-IP</code> for your service). If this is the case, DNS caching should not be an issue, because that single virtual IP should be splitting traffic among the actual backends.</p>

<p>Try <code>kubectl get endpoints your-service</code> to confirm that your service actually knows about all of your backends.</p>

<p>If you have a <a href="http://kubernetes.io/docs/user-guide/services/#headless-services" rel="nofollow">headless service</a>, a DNS lookup will return an A record with 10 IPs (one for each of your Pods). If your client is always choosing the first IP in an A record, that would also explain the behavior you're seeing.</p>

python http dns kubernetes grpc