Experience Case Study for Packet Loss Issue on PIS System of a Metro Network.

X metro line's PIS system uses multicast service and the same VLAN for different trains, forming a large layer 2 network.

During high-speed train operation, the control center server experienced severe packet loss when pinging the onboard AP. In the early stages of operation, there was no packet loss, and there were no issues when configured on the Y line.

There is no packet loss in the early deployment of the X-line, so the configuration itself should not be a big problem. Considering that it is a large layer 2 network, it may be that the traffic model has changed. During the EMU test, I checked the signal strength established by the mesh link and the change of air interface interference by brushing the script, and found that the packet loss of the on-board AP was accompanied by a surge in the air interface. Check the diagnosis of the trackside AP and find that the proportion of broadcast packets on the AP's wired port is abnormal, and the inbound traffic is very large.

================display interface================

GigabitEthernet1/0/1 current state: UP

Last 300 seconds input: 3373 packets/sec 1281297 bytes/sec 1%

Last 300 seconds output: 2 packets/sec 407 bytes/sec 0%

Input (total): 213199458 packets, 167740424629 bytes

          1186882 unicasts, 87101754 broadcasts, 124910822 multicasts, - pauses

It is suspected that the vehicle-to-ground communication is interrupted due to abnormal broadcast packet traffic in the large layer 2 network, because broadcast packets are sent at the lowest rate on the wireless air interface, occupying air interface resources for a long time, increasing the air interface, and affecting wireless MESH communication

In order to verify the conjecture, port mirroring was performed on the access switch connected to the trackside AP, and the packet captured by wireshark was analyzed. It was found that 70%+ of the packets were arp packets, and the ARP rate could reach 2000pps. The source mac of a large number of arp was the control A server in the center.

After removing the server that was sending a large number of ARP requests from the network, wireless communication between the train and ground was restored to normal.