Kubernetes NetworkAttachmentDefinition

Question

currently I'm running a Kubernetes Cluster.

Host Network: 10.17.20.x Docker Network: 172.17.60.x

Im running a RabbitMQ pod which has the IP: 172.17.60.217 Annotations:

• cni.projectcalico.org/podIP: 172.17.60.217/32 cni.projectcalico.org/podIPs: 172.17.60.217/32

All my pods need to be connected to RabbitMQ. This works quite fine. I need to run pod "A" as macvlan cause it needs to be reachable from the Hostnetwork for peripherie.

So from the topology point of view. My pod "A" need to have two Network interfaces to be able to connect to rabbitmq 172.17.60.217 and get a Hostnetwork IP assigned. Therefore a came up with NetworkAttachmentDefinitions. My Plan was to create a NetworkAttachment to assign a IP from the Hostnetwork.

NetworkAttachmentDefinition:
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: macvlan-conf
spec:
  config: '{
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "type": "macvlan",
          "capabilities": { "ips": true },
          "master": "ens224",
          "mode": "bridge",
          "ipam": {
            "type": "static",
            "routes": [
              {
                "dst": "0.0.0.0/0",
                "gw": "10.17.20.1"
              }
            ]
          }
        }
      ]
    }'

It seems that the IP's are assigned correct.

apiVersion: v1
kind: Pod
metadata:
  labels:
    run: udpechroute
  name: udpechoroute
  annotations:
  annotations:
    k8s.v1.cni.cncf.io/networks: '[ {
      "name": "macvlan-conf",
      "ips": [ "10.17.20.124/24" ],
      "route": [ "10.17.20.1" ]
      }]'
spec:
  containers:
  - image: alpine/socat:latest
    imagePullPolicy: Never
    name: udpecho
    args:
      - "-v"
      - "PIPE"
      - "udp-recvfrom:5553,fork"
  restartPolicy: Always
status: {}

kubectl describe pod udpechoroute

Name:         udpechoroute
Namespace:    kube-system
Priority:     0
Node:         openstack1/10.17.20.21
Start Time:   Fri, 26 Nov 2021 17:38:10 +0100
Labels:       run=udpechroute
Annotations:  cni.projectcalico.org/podIP: 172.17.60.206/32
              cni.projectcalico.org/podIPs: 172.17.60.206/32
              k8s.v1.cni.cncf.io/network-status:
                [{
                    "name": "",
                    "ips": [
                        "172.17.60.206"
                    ],
                    "default": true,
                    "dns": {}
                },{
                    "name": "kube-system/macvlan-conf",
                    "interface": "net1",
                    "ips": [
                        "10.17.20.124"
                    ],
                    "mac": "26:7f:a1:40:79:c9",
                    "dns": {}
                }]
              k8s.v1.cni.cncf.io/networks: [ { "name": "macvlan-conf", "ips": [ "10.17.20.124/24" ], "route": [ "10.17.20.1" ] }]
              k8s.v1.cni.cncf.io/networks-status:
                [{
                    "name": "",
                    "ips": [
                        "172.17.60.206"
                    ],
                    "default": true,
                    "dns": {}
                },{
                    "name": "kube-system/macvlan-conf",
                    "interface": "net1",
                    "ips": [
                        "10.17.20.124"
                    ],
                    "mac": "26:7f:a1:40:79:c9",
                    "dns": {}
                }]

[root@openstack1]# kubectl exec -it udpechoroute /bin/sh -- route

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         169.254.1.1     0.0.0.0         UG    0      0        0 eth0
10.17.20.0      *               255.255.255.0   U     0      0        0 net1
169.254.1.1     *               255.255.255.255 UH    0      0        0 eth0

[root@openstack1]# kubectl exec -it udpechowithoutan /bin/sh -- route

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         169.254.1.1     0.0.0.0         UG    0      0        0 eth0
169.254.1.1     *               255.255.255.255 UH    0      0        0 eth0

My Issue: I delpyoed a Pod which is not using any NetworkAttachmentDefinition so to know the Gateway information for the pod. But from my udpechoroute Pod I'm able to ping the RabbitMQ adresse but it cannot be reached from the peripherie on IP 10.17.20.124.

kubectl exec -it udpechoroute -- ping 172.17.60.217:

PING 172.17.60.217 (172.17.60.217): 56 data bytes
64 bytes from 172.17.60.217: seq=0 ttl=63 time=0.164 ms
64 bytes from 172.17.60.217: seq=1 ttl=63 time=0.131 ms
64 bytes from 172.17.60.217: seq=2 ttl=63 time=0.106 ms

From 10.17.20.x the Ping to 10.17.20.124 doesn't work.

Pinging 10.17.20.124 with 32 bytes of data:
Request timed out.

No Firewall rules are blocking the communication.

Do you have any idea what i did wrong?