★ Multiple virtual machines restarting irregularly

UIS/CAS

Multiple virtual machines restarting irregularly

1.   Check /var/log/libvirt/qemu/vm_name.log to see if the reason for the virtual machine restart is a crash.

Take VM “RHEL 7.9_4” as an example,  2025-11-18 12:33 , UIS web report this VM
restarted because of fault.     

2025-11-18 12:31:50.336+0000: shutting down, reason=crashed

2. The virtual machine "RHEL 7.9_4" runs on the node HCI-Node-01. Checking the host log: /var/log/message, you can see a warning about insufficient cgroup memory at the same time. This means that the virtual machine process triggered the memory usage control group threshold, causing the process to be terminated abnormally.
Nov 18 12:31:49 HCI-Node-01 kernel:[89580.599157] oom-kill:cOnstraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=emulator,mems_allowed=0-1,oom_memcg=/machine.slice,task_memcg=/machine.slice/machine-qemu\x2d33\x2dRHEL7\x2d9\x2d4.scope,task=kvm,pid=2016872,uid=0
Nov 18 12:31:49 HCI-Node-01 kernel: [89580.599207] Memory cgroup
out of memory: Killed process 2016872 (kvm)total-vm:5556908kB, anon-rss:1978036kB, file-rss:31684kB,
shmem-rss:4kB, UID:0 pgtables:4552kB oom_score_adj:400
Nov 18 12:31:50 HCI-Node-01 runlevel[2812933]: [runlevel] called by PID 2812932 (/bin/bash/var/lib/ceph/shell/hbm_protect.sh)
 
3. As shown in the figure below, you can see the records of the virtual machine processes running on these two nodes being terminated due to insufficient memory in the control group. This record and the virtual machine recorded in the operation log.

You can check the /etc/cgconfig.conf file to obtain the memory.limit_in_bytes parameter, which is the memory limit of the cgroup.

If a batch of tasks is executed in the virtual machine within a period of time, the memory used by the virtual machine will increase. Cgroup triggering is calculated based on the actual physical memory used by the kvm process of the virtual machine. Since the kvm process itself also needs to consume resources, the kvm process of the virtual machine usually consumes more memory than the memory usage displayed inside the virtual machine. Therefore, when configuring memory, a portion of memory should be reserved; otherwise, the virtual machine may be terminated randomly. 

1. Increase the memory allocation of the host;

2. Migrate memory-intensive virtual machines to another host with sufficient memory resources.

Supplementary knowledge point: cgroup

In KVM (Kernel-based Virtual Machine) virtualization, cgroups (Control Groups) re primarily used for esource isolation, limitation, and accounting. They ensure that virtual machines (VMs) do not over-consume host resources, thereby guaranteeing system stability and performance.