Apr 2021

aka Centos Emergency Mode.

This has been listed under IPHE, not because it is specific to IPHE (because it is not) but because it was on an IPHE system that I first observed this phenomenon.

After a server was rebooted (and I cannot say whether this was a reboot from a command line, a server crash, someone pulling out the power cables, a power cut etc) the end result is that the server will not fully boot up, instead we end up in 'emergency mode'.

When logging in to the server from command KVM or from BMC (or whatever remote management is present) we can see the following tell tale screen:



To move past this page you have to enter Control-D to get to the login prompt and complete the boot up process. This is a hassle to do every time there is a reboot. A clue of what to look at is in the text, the command journalctl -xb can be executed once you have logged in.

On my system this file was 30,888 lines long, so hundreds of pages long. After searching through it though I did find this section:

-- The start-up result is done.
Apr 20 17:04:10 k8master1 kernel: power_meter ACPI000D:00: Found ACPI power meter.
Apr 20 17:04:10 k8master1 systemd-fsck[19721]: /dev/mapper/vg_main-lv_var: Inodes that were part of a corrupted orphan linked list found.
Apr 20 17:04:10 k8master1 systemd-fsck[19721]: /dev/mapper/vg_main-lv_var: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
Apr 20 17:04:10 k8master1 systemd-fsck[19721]: (i.e., without -a or -p options)
Apr 20 17:04:10 k8master1 systemd-fsck[19721]: fsck failed with error code 4.
Apr 20 17:04:10 k8master1 systemd-fsck[19721]: Running request emergency.target/start/replace
Apr 20 17:04:10 k8master1 systemd[1]: Started File System Check on /dev/mapper/vg_main-lv_var.
-- Subject: Unit systemd-fsck@dev-mapper-vg_main\x2dlv_var.service has finished start-up

Apr 20 17:04:10 k8master1 systemd-fsck[19721]: /dev/mapper/vg_main-lv_var: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
so /dev/mapper/vg_main-lv_var has an inode issue