K8s pods unable to reach external VM via internal IP

ghz 1years ago ⋅ 7586 views

Question

I am migrating to Azure platform from GCP. I have a k8s cluster that needs to talk to external Cassandra cluster using internal IP(s), in the same Azure region but different VNET. I have the VNET(s) peered. I can reach the Cassandra cluster from the K8s nodes and vice versa but cannot reach them from the pods.

This seems to be some Azure networking issue. I have opened up firewall rules for the pods to reach Cassandra but with no luck. How best should I solve this?

Answer

Because Azure can't find your private IP address of your pods. We can use Azure route table to connect them.

Here is my test, two resource group, one for k8s and another one for a signal VM.

Here is the information about pods:

root@k8s-master-CA9C4E39-0:~# kubectl get pods --output=wide
NAME                       READY     STATUS    RESTARTS   AGE       IP             NODE
influxdb                   1/1       Running   0          59m       10.244.1.166   k8s-agent-ca9c4e39-0
my-nginx-858393261-jrz15   1/1       Running   0          1h        10.244.1.63    k8s-agent-ca9c4e39-0
my-nginx-858393261-wbpl6   1/1       Running   0          1h        10.244.1.62    k8s-agent-ca9c4e39-0
nginx                      1/1       Running   0          52m       10.244.1.179   k8s-agent-ca9c4e39-0
nginx3                     1/1       Running   0          43m       10.244.1.198   k8s-agent-ca9c4e39-0

The information about K8s agent and master :

enter image description here

The information about the signal VM:

enter image description here

By default, we can't use 172.16.0.4 to ping 10.244.1.0/24. We should add an Azure route table, then we can ping that pod IP address:

enter image description here

Here is my result:

root@jasonvm2:~# ping 10.244.1.166
PING 10.244.1.166 (10.244.1.166) 56(84) bytes of data.
64 bytes from 10.244.1.166: icmp_seq=1 ttl=63 time=2.61 ms
64 bytes from 10.244.1.166: icmp_seq=2 ttl=63 time=1.42 ms
--- 10.244.1.166 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 1.424/2.019/2.614/0.595 ms
root@jasonvm2:~# ping 10.244.1.166
PING 10.244.1.166 (10.244.1.166) 56(84) bytes of data.
64 bytes from 10.244.1.166: icmp_seq=1 ttl=63 time=2.56 ms
64 bytes from 10.244.1.166: icmp_seq=2 ttl=63 time=1.10 ms
^C
--- 10.244.1.166 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 1.102/1.833/2.564/0.731 ms
root@jasonvm2:~# ping 10.244.1.63
PING 10.244.1.63 (10.244.1.63) 56(84) bytes of data.
64 bytes from 10.244.1.63: icmp_seq=1 ttl=63 time=2.89 ms
64 bytes from 10.244.1.63: icmp_seq=2 ttl=63 time=2.27 ms
--- 10.244.1.63 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 2.271/2.581/2.892/0.314 ms

About Azure route table, please refer to this [link](https://learn.microsoft.com/en-us/azure/virtual-network/virtual- networks-udr-overview).

#kubernetes