Network Topology
Devices and Versions: S12508X-AF R2820,S6860-54HF F2707
Networking: Two S6860 devices are used as a group of leaf devices, and two S12508X-AF devices are used as spine/border devices. Tunnels are established between the leaf and spine/border devices, and all terminals under the leaf device are single-hanging. The following figure shows a partial network diagram.
Problem Description
In the on-site M-LAG+EVPN networking, two S6860 devices are used as leaf devices, and two 12508X-AF devices are used as spine/border devices. It is found that the single-hanging terminals under the leaf cannot access the external network. The ARP and MAC table learning on the leaf device is normal. When capturing packets when the terminal pings the external network, the ICMP packets are sent from the leaf device, and the spine/border device also receives the ICMP packets but does not forward them upstream, confirming that the packet loss occurs on the spine/border device.
Process Analysis
The spine/border device does not forward the received packets. We first suspect whether the packets sent from the leaf are encapsulated correctly. From the captured packets, the packets are normal. The outer layer of the VXLAN has the source and destination IP addresses of the tunnel, and the inner layer has the source and destination IP addresses of the original packet:
Since there is no problem with the packets, why doesn't the spine/border device correctly decapsulate and forward the packets? When a device receives an encapsulated packet, if the encapsulated source and destination IP addresses can find a symmetric tunnel on the device, it can decapsulate the packet; otherwise, it cannot decapsulate and forward the packet normally. We found that the spine/border device only has the following tunnel based on the leaf device:
source 60.1.1.5, destination 60.1.1.251
The captured packets are also encapsulated with the source and destination IP addresses of this tunnel;
On the spine/border device, there are two tunnels from the leaf:
source 60.1.1.251, destination 60.1.1.1
source 60.1.1.251, destination 60.1.1.2
Obviously, neither of these two tunnels on the spine/border device is the symmetric tunnel for encapsulating and decapsulating the packets, so the spine/border device cannot correctly decapsulate and forward the packets.
Why are the tunnels on the leaf and spine/border devices asymmetric? Further inspection of the configuration information reveals that 60.1.1.1 and 60.1.1.2 are the loopback 0 addresses of the two leaf devices, and 60.1.1.5 and 60.1.1.251 are the group IPs of the leaf and spine/border devices. And because there is a single-hanging terminal under the leaf, the leaf device is configured with the evpn drni local local-ip remote remote-ip command; and the spine/border device introduces 5-class routes and configures the nexthop evpn-drni group-address command:
In this configuration, since all terminals are single-hanging under the leaf, the leaf device releases 3-class routes with the group IP and 2-class routes with the local IP, and the spine/border device only imports 5-class routes published by the group IP. After receiving the 5-class routes published by the spine/border device, the leaf device establishes a virtual address tunnel with the source and destination IP addresses of 60.1.1.5 and 60.1.1.251, respectively; the spine/border device receives 3-class and 2-class routes published by the leaf, but since the spine/border device does not have a VSI configuration, it cannot receive 3-class routes and can only establish a tunnel to the actual address of the leaf device based on the next hop of the 2-class routes.
From the above analysis, it can be seen that in the on-site network configuration, the tunnels between the leaf and spine/border devices are not symmetric, and the spine/border device cannot correctly decapsulate the packets, resulting in the fault phenomenon on-site. So how should we solve it?
Solution
Configure vxlan default-decapsulation source interface on the spine/border device to force decapsulation and solve the problem. After configuring this command, when the spine/border device receives packets encapsulated with VXLAN, it will directly force decapsulation and then proceed with the normal forwarding process.