Clients that connected to S5130S have very low speed on Internet

S5130S-HI as Access switch 

Client----- S5130S-HI---- Core ---- External network

Customer told us that their clients which connected to S5130S-HI have low internet speed

1. We troubleshooted the Layer 2 network and found that there are no loops, stp is not blocking, and the interface bandwidth is normal.

2. We check the gateway diagnostics. Client traffic were sent to the gateway, and then forward to Internet based on the layer 3. Check the logbuff of the device, there is a log of http packets forwarding over speed limit of CPU. Generally, there should be no http-related service on the device, and http messages should normally be forwarded by hardware without cpu processing. Therefore, we suspect that the device routing specification is insufficient to cause the problem.

        1566: PktType=HTTP/HTTPS, SrcMAC=6887-c69c-72d7, Dropped from interface=Ten-GigabitEthernet1/0/52 at Stage=9, StageCnt=72, TotalCnt=3438, MaxRateInterface=Ten-GigabitEthernet1/0/52.

        1728: PktType=HTTP/HTTPS, SrcMAC=6887-c69c-b427, Dropped from interface=Ten-GigabitEthernet2/0/52 at Stage=1, StageCnt=22, TotalCnt=3052, MaxRateInterface=Ten-GigabitEthernet2/0/52.

        12886: PktType=HTTP/HTTPS, SrcMAC=6887-c69c-72d7, Dropped from interface=Ten-GigabitEthernet1/0/52 at Stage=9, StageCnt=72, TotalCnt=3438, MaxRateInterface=Ten-GigabitEthernet1/0/52.

        13048: PktType=HTTP/HTTPS, SrcMAC=6887-c69c-b427, Dropped from interface=Ten-GigabitEthernet2/0/52 at Stage=1, StageCnt=22, TotalCnt=3052, MaxRateInterface=Ten-GigabitEthernet2/0/52.

        13204: PktType=HTTP/HTTPS, SrcMAC=6887-c69c-72d7, Dropped from interface=Ten-GigabitEthernet1/0/52 at Stage=1, StageCnt=18, TotalCnt=3456, MaxRateInterface=Ten-GigabitEthernet1/0/52. 

3. We looked at the local logbuff on the device and found the following error

Aug 28 2023 17:43:58:0186:

LINE:2171-TASK:kfib/1-FUNC:drv_ipv4_uc_hard_addroute:

drv_l3uc_cpss_add_ipv4_lpm return 0x4001000b //which means over the limit

Aug 28 2023 17:43:58:0186:

LINE:4085-TASK:kfib/1-FUNC:drv_l3uc_cpss_add_ipv4_lpm:

Fail to add deep!Unit=0,iRv=20,ip=0x23f67c0a,mask=0xffffffff,vrf=0,intf=99.

4. Routing overspecification will lead to routing down to the hardware. Therefore, it can not be forwarded by hardware, it will be forwarded by cpu. But cpu forwarding will have a speed limit, resulting in packet loss, http packet loss retransmission will have access to the web page lag.

  ===============debug rxtx softcar show slot 1=============== 

ID  Type                RcvPps Rcv_All    DisPkt_All Pps  Dyn Swi Hash Am APps

63  HTTP/HTTPS          45     5555101    3468       500  S   On  SMAC 8 512

  ===============debug rxtx softcar show slot 2=============== 

ID  Type                RcvPps Rcv_All    DisPkt_All Pps  Dyn Swi Hash Am APps

63  HTTP/HTTPS          44     5492449    3063       500  S   On  SMAC 8 512

5. The maximum specification of arp + routes for this device is only 1000, which is a relatively small specification. The device at the site is ospf-enabled and has a large amount of routes, so it is recommended to scale down the routes or use a static default route for Layer 3 access.

Routes are oversized and need to be shrunk down or use static default routes for Layer 3 access.