Both sides previous revisionPrevious revisionNext revision | Previous revision |
centos_not_booting [2021/04/21 18:16] – walkeradmin | centos_not_booting [2023/03/09 22:35] (current) – external edit 127.0.0.1 |
---|
**<color #ed1c24>aka Centos Emergency Mode.</color>** | **<color #ed1c24>aka Centos Emergency Mode.</color>** |
---- | ---- |
| ==== Introduction ==== |
\\ | \\ |
This has been listed under IPHE, not because it is specific to IPHE (because it is not) but because it was on an IPHE system that I first observed this phenomenon. | This has been listed under IPHE, not because it is specific to IPHE (because it is not) but because it was on an IPHE system that I first observed this phenomenon. |
\\ | \\ |
\\ | \\ |
| |
| ---- |
| ==== The Cause ==== |
To move past this page you have to enter <color #ed1c24>Control-</color>D to get to the login prompt and complete the boot up process. This is a hassle to do every time there is a reboot. A clue of what to look at is in the text, the command j<color #ed1c24>ournalctl -xb</color> can be executed once you have logged in. | To move past this page you have to enter <color #ed1c24>Control-</color>D to get to the login prompt and complete the boot up process. This is a hassle to do every time there is a reboot. A clue of what to look at is in the text, the command j<color #ed1c24>ournalctl -xb</color> can be executed once you have logged in. |
\\ | \\ |
\\ | \\ |
\\ | \\ |
Apr 20 17:04:10 k8master1 systemd-fsck[19721]: /dev/mapper/vg_main-lv_var: <color #ed1c24>Inodes</color> that were part of a corrupted orphan linked list found. | Apr 20 17:04:10 k8master1 systemd-fsck[19721]: <color #ed1c24>/dev/mapper/vg_main-lv_var</color>: <color #ed1c24>Inodes</color> that were part of a corrupted orphan linked list found. |
\\ | \\ |
\\ | \\ |
So /dev/mapper/vg_main-lv_var has an <color #ed1c24>Inode</color> issue. | So <color #ed1c24>/dev/mapper/vg_main-lv_var</color> has an <color #ed1c24>Inode</color> issue. An <color #ed1c24>Inode</color> is a reference that points to each file on the disk, there is an <color #ed1c24>Inode</color> for each file, and for empty space also. The <color #ed1c24>Inodes</color> should all line up nicely, but in the case of a server crash or sudden power loss, the process writing to the disk may not have time to update all the <color #ed1c24>Inodes</color>, and so a discrepancy occurs on the disk that the system notices. |
| \\ |
| \\ |
| Each time the server boots, this discrepancy causes the server to go in to <color #ed1c24>Emergency Mode</color>. On a disk that is being written to a lot, this is not an uncommon occurrence, there are two choices, run the <color #ed1c24>fsck</color>, which may not be possible (especially remotely) or tell the system to not perform the check at boot, thus avoiding going in to <color #ed1c24>Emergency Mode</color> while rebooting. |
| \\ |
| \\ |
| |
| ---- |
| |
| ==== The Remedy ==== |
| \\ |
| We need to tell the system to not perform the <color #ed1c24>fsck</color> at boot, and to do this we need to edit the file <color #ed1c24>fstab</color> in the <color #ed1c24>etc</color> directory (this is for CentOS).\\ |
| \\ |
| \\ |
| Open the file in a text editor by using <color #ed1c24>vi /etc/fstab</color> |
| \\ |
| <file> |
| # |
| # /etc/fstab |
| # Created by anaconda on Thu Nov 5 18:45:24 2020 |
| # |
| # Accessible filesystems, by reference, are maintained under '/dev/disk' |
| # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info |
| # |
| /dev/mapper/vg_main-lv_root / ext4 defaults 1 1 |
| UUID=ce253ff9-86ee-480b-87b3-7efa803bedda /boot ext4 defaults 1 2 |
| /dev/mapper/vg_main-lv_opt /opt ext4 defaults 1 2 |
| /dev/mapper/vg_main-lv_var /var ext4 defaults 1 1 |
| /dev/mapper/vg_main-lv_applogs /applogs ext4 defaults 1 2 |
| </file> |
| Above we see the contents of the <color #ed1c24>fstab</color> file (your own <color #ed1c24>fstab</color> will differ) and we know that the <color #ed1c24>journalctl -xe</color> was complaining about<color #ed1c24> /dev/mapper/vg_main-lv_var</color> |
| \\ |
| \\ |
| Each entry in the fstab file has six fields: |
| <file> |
| 1 2 3 4 5 6 |
| |
| /dev/mapper/vg_main-lv_var /var ext4 defaults 1 1 |
| </file> |
| \\ |
| It is the last number we want to change, and we will change this to a zero, this will mark this partition (and only this partition) to ignore fsck at boot. |
| \\ |
| <file> |
| So we will change: |
| |
| /dev/mapper/vg_main-lv_var /var ext4 defaults 1 1 |
| |
| to |
| |
| /dev/mapper/vg_main-lv_var /var ext4 defaults 1 0 |
| </file> |
| Save this file, and now you should be able to do a test reboot, and the system should boot normally to user login page rather than emergency mode. |
| \\ |
| \\ |
| \\ |