Differences

This shows you the differences between two versions of the page.

--- centos_not_booting [2021/04/21 17:51] – walkeradmin
+++ centos_not_booting [2023/03/09 22:35] (current) – external edit 127.0.0.1
@@ Line 5: / Line 5: @@
 **<color #ed1c24>aka Centos Emergency Mode.</color>**
 ----
+==== Introduction ====
 \\
 This has been listed under IPHE, not because it is specific to IPHE (because it is not) but because it was on an IPHE system that I first observed this phenomenon.
@@ Line 14: / Line 15: @@
 When logging in to the server from command KVM or from BMC (or whatever remote management is present) we can see the following tell tale screen:
 \\
-{{:1056.png?400|}}
+\\
+{{ :1056.png?800 |}}
+\\
+\\
+----
+==== The Cause ====
+To move past this page you have to enter <color #ed1c24>Control-</color>D to get to the login prompt and complete the boot up process. This is a hassle to do every time there is a reboot. A clue of what to look at is in the text, the command j<color #ed1c24>ournalctl -xb</color> can be executed once you have logged in.
+\\
+\\
+On my system this file was 30,888 lines long, so hundreds of pages long. After searching through it though I did find this section:
+\\
+<file>
+-- The start-up result is done.
+Apr 20 17:04:10 k8master1 kernel: power_meter ACPI000D:00: Found ACPI power meter.
+Apr 20 17:04:10 k8master1 systemd-fsck[19721]: /dev/mapper/vg_main-lv_var: Inodes that were part of a corrupted orphan linked list found.
+Apr 20 17:04:10 k8master1 systemd-fsck[19721]: /dev/mapper/vg_main-lv_var: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
+Apr 20 17:04:10 k8master1 systemd-fsck[19721]: (i.e., without -a or -p options)
+Apr 20 17:04:10 k8master1 systemd-fsck[19721]: fsck failed with error code 4.
+Apr 20 17:04:10 k8master1 systemd-fsck[19721]: Running request emergency.target/start/replace
+Apr 20 17:04:10 k8master1 systemd[1]: Started File System Check on /dev/mapper/vg_main-lv_var.
+-- Subject: Unit systemd-fsck@dev-mapper-vg_main\x2dlv_var.service has finished start-up
+</file>
+\\
+The important parts are:
+\\
+\\
+Apr 20 17:04:10 k8master1 systemd-fsck[19721]: /dev/mapper/vg_main-lv_var: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
+\\
+\\
+The partition <color #ed1c24>vg_main-lv_var</color> has an issue, and Linux wants us to run <color #ed1c24>fsck</color>. <color #ed1c24>fsck</color> is a file check tool, however it will not run on mounted file systems, and the only way to unmount them is if you boot from say a USB stick.
+\\
+\\
+The other important part is:
+\\
+\\
+Apr 20 17:04:10 k8master1 systemd-fsck[19721]: <color #ed1c24>/dev/mapper/vg_main-lv_var</color>: <color #ed1c24>Inodes</color> that were part of a corrupted orphan linked list found.
+\\
+\\
+So <color #ed1c24>/dev/mapper/vg_main-lv_var</color> has an <color #ed1c24>Inode</color> issue. An <color #ed1c24>Inode</color> is a reference that points to each file on the disk, there is an <color #ed1c24>Inode</color> for each file, and for empty space also. The <color #ed1c24>Inodes</color> should all line up nicely, but in the case of a server crash or sudden power loss, the process writing to the disk may not have time to update all the <color #ed1c24>Inodes</color>, and so a discrepancy occurs on the disk that the system notices.
+\\
+\\
+Each time the server boots, this discrepancy causes the server to go in to <color #ed1c24>Emergency Mode</color>. On a disk that is being written to a lot, this is not an uncommon occurrence, there are two choices, run the <color #ed1c24>fsck</color>, which may not be possible (especially remotely) or tell the system to not perform the check at boot, thus avoiding going in to <color #ed1c24>Emergency Mode</color> while rebooting.
+\\
+\\
+----
+==== The Remedy ====
+\\
+We need to tell the system to not perform the <color #ed1c24>fsck</color> at boot, and to do this we need to edit the file <color #ed1c24>fstab</color> in the <color #ed1c24>etc</color> directory (this is for CentOS).\\
+\\
+\\
+Open the file in a text editor by using <color #ed1c24>vi /etc/fstab</color>
+\\
+<file>
+#
+# /etc/fstab
+# Created by anaconda on Thu Nov  5 18:45:24 2020
+#
+# Accessible filesystems, by reference, are maintained under '/dev/disk'
+# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
+#
+/dev/mapper/vg_main-lv_root /                       ext4    defaults        1 1
+UUID=ce253ff9-86ee-480b-87b3-7efa803bedda /boot                   ext4    defaults        1 2
+/dev/mapper/vg_main-lv_opt /opt                    ext4    defaults        1 2
+/dev/mapper/vg_main-lv_var /var                    ext4    defaults        1 1
+/dev/mapper/vg_main-lv_applogs /applogs            ext4    defaults        1 2
+</file>
+Above we see the contents of the <color #ed1c24>fstab</color> file (your own <color #ed1c24>fstab</color> will differ) and we know that the <color #ed1c24>journalctl -xe</color> was complaining about<color #ed1c24> /dev/mapper/vg_main-lv_var</color>
+\\
+\\
+Each entry in the fstab file has six fields:
+<file>
+                          2                       3       4               5 6
+/dev/mapper/vg_main-lv_var /var                    ext4    defaults        1 1
+</file>
+\\
+It is the last number we want to change, and we will change this to a zero, this will mark this partition (and only this partition) to ignore fsck at boot.
+\\
+<file>
+So we will change:
+/dev/mapper/vg_main-lv_var /var                    ext4    defaults        1 1
+to
+/dev/mapper/vg_main-lv_var /var                    ext4    defaults        1 0
+</file>
+Save this file, and now you should be able to do a test reboot, and the system should boot normally to user login page rather than emergency mode.
+\\
+\\
 \\