CAS management web page access abnormal performance, data refresh is slow

Version：CAS E0530H12

Access to CVM page is abnormal, the performance data of virtual machine interface is very slow to refresh, the storage pool page has been circling, and the overall operation of the platform is stuck.

1. First of all, excluding the network and environmental factors: check whether the client is two-layer access or cross-three-tier, whether there is a problem with the client computer, whether the phenomenon of trying to access with other PCs have the same problem;

2. Check the number of Tomcat connections: ssh to CVM node ,then use command netstat - NAT | grep 8080 | WC - L. If it turns out the number of connections is too large, then may cause the page load be stuck. In normal environment, the number of connections is about 200, which is normal and has no fixed value;

3. Check whether the storage is normal:Perform iostat to check the storage delay and busy degree. It is found that the% util of dm-0 storage volume reaches 100%, the disk is busy, there are many I / O requests, and it runs at full load. 

4. Confirm whether there is cluster lock :ssh to cvk shell line and execute OCFS2_ kernel_ stat_ collect.sh command to check whether there is a hung up process.As shown in the figure below, it is found that there is a hung up process and dm-0 is busy printing. A similar log file named cvk01_ ocfs2_ kernel_ state_ 202005231721.log is generated in the / var / log directory. The log file contains the following logs. It is preliminarily judged that IO is blocked

5. Check the syslog log log:there are a large number of links concussion, multi-path connection failure print, so we can judge the storage failure

6. Multipath - ll view multipath, with failed print

Check the storage and storage link conditions as soon as possible according to the corresponding storage manufacturer, such as optical power, packet loss, etc., and replace the HBA card or optical fiber, optical port, etc. if necessary.