OS/Kernel logging - system freeze/hung
We are encountering an issue where a server becomes completely unresponsive, with no clear indication of what caused the hang. example:
/var/log/syslog:
/var/log/kern.log
As shown above, there are no log entries between 15:45:02 and the subsequent boot at 20:38:14. During this period, the node was completely unresponsive and we were unable to SSH into the server, and there was no useful information visible on the console.
After the reboot, we were also unable to find any relevant messages in the system logs that would help identify the cause of the hang.
we do have following setting:
Given that kernel.hung_task_panic=1, we expected to see a hung-task stack trace or kernel panic information if tasks were blocked for longer than the configured timeout. However, no such information was captured in the logs.
i am primarily looking to understand if there additional boot/kernel settings ( sysctl) or debugging mechanisms (SysRq, NMI watchdog, kdump, netconsole, serial console, etc.) that should be enabled to capture diagnostic information during such hangs?
also, under what circumstances a system can become completely unresponsive without generating hung-task traces or panic logs?
this issue occured with 3 out of 8 servers today within a span of 1 hour.