CentOS Not Booting

Apr 2021
aka Centos Emergency Mode.

This has been listed under IPHE, not because it is specific to IPHE (because it is not) but because it was on an IPHE system that I first observed this phenomenon.
After a server was rebooted (and I cannot say whether this was a reboot from a command line, a server crash, someone pulling out the power cables, a power cut etc) the end result is that the server will not fully boot up, instead we end up in 'emergency mode'.
When logging in to the server from command KVM or from BMC (or whatever remote management is present) we can see the following tell tale screen:
To move past this page you have to enter Control-D to get to the login prompt and complete the boot up process. This is a hassle to do every time there is a reboot. A clue of what to look at is in the text, the command journalctl -xb can be executed once you have logged in.
On my system this file was 30,888 lines long, so hundreds of pages long. After searching through it though I did find this section:

-- The start-up result is done.
Apr 20 17:04:10 k8master1 kernel: power_meter ACPI000D:00: Found ACPI power meter.
Apr 20 17:04:10 k8master1 systemd-fsck[19721]: /dev/mapper/vg_main-lv_var: Inodes that were part of a corrupted orphan linked list found.
Apr 20 17:04:10 k8master1 systemd-fsck[19721]: /dev/mapper/vg_main-lv_var: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
Apr 20 17:04:10 k8master1 systemd-fsck[19721]: (i.e., without -a or -p options)
Apr 20 17:04:10 k8master1 systemd-fsck[19721]: fsck failed with error code 4.
Apr 20 17:04:10 k8master1 systemd-fsck[19721]: Running request emergency.target/start/replace
Apr 20 17:04:10 k8master1 systemd[1]: Started File System Check on /dev/mapper/vg_main-lv_var.
-- Subject: Unit systemd-fsck@dev-mapper-vg_main\x2dlv_var.service has finished start-up

The important parts are:
Apr 20 17:04:10 k8master1 systemd-fsck[19721]: /dev/mapper/vg_main-lv_var: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
The partition vg_main-lv_var has an issue, and Linux wants us to run fsck. fsck is a file check tool, however it will not run on mounted file systems, and the only way to unmount them is if you boot from say a USB stick.
The other important part is:
Apr 20 17:04:10 k8master1 systemd-fsck[19721]: /dev/mapper/vg_main-lv_var: Inodes that were part of a corrupted orphan linked list found.
So /dev/mapper/vg_main-lv_var has an Inode issue. An Inode is a reference that points to each file on the disk, there is an Inode for each file, and for empty space also. The Inodes should all line up nicely, but in the case of a server crash or sudden power loss, the process writing to the disk may not have time to update all the Inodes, and so a discrepancy occurs on the disk that the system notices.
Each time the server boots, this discrepancy causes the server to go in to Emergency Mode. On a disk that is being written to a lot, this is not an uncommon occurrence, there are two choices, run the fsck, which may not be possible (especially remotely) or tell the system to not perform the check at boot, thus avoiding going in to Emergency Mode while rebooting.