Kubernetes K8

Apr 2021

Overview

Containers contain functions for software components, a container can be a web server, a data processor or an data I/O device, anything really that you used to be able to run on a bare metal server or a VM. Containers can run on multiple servers in multiple locations.
Kubernetes provides an open-source API that controls how and where those containers will run, and manages continer failure and restarts.

Kubernetes Commands

There are many Kubernets commands, here we will explore some that will be useful when dealing with MediaKind HeadEnd solutions (IPHE).

get ns

kubectl -n mediakind get nodes

PODs run in a particular namespace, there can be more than one namespace in a system. To be able to use any commands you have to know which namespace you wish to use, and specify that namespace in your command line.
Use the following command to show available namespace.

kubectl get ns
NAME STATUS AGE
default Active 78d
kube-public Active 78d
kube-system Active 78d
mediakind Active 78d

get nodes

kubectl -n mediakind get nodes

In the above example, kubectl is the command, mediakind is the namespace we are dealing with and get nodes is the parameters for the command.

NAME STATUS ROLES AGE VERSION
172.25.27.17 Ready master,node 20d v1.13.12
172.25.27.20 Ready master,node 20d v1.13.12
172.25.27.21 Ready master,node 20d v1.13.12
172.25.27.26 Ready node 20d v1.13.12

kubectl -n mediakind get nodes returns the number of 'servers' in the deployment, these nodes can be bare metal or Virtual Machines.

describe node

kubectl -n mediakind describe node ipaddress

Above the command is describing a particular node in the system, the node IP address was derived by using the get node command.

kubectl -n mediakind describe node 172.25.27.26
Name: 172.25.27.26
Roles: node
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
encoding-hd-services=enabled
encoding-ott-services=enabled
encoding-sd-services=enabled
kubernetes.io/hostname=172.25.27.26
mux-bckp=enabled
node-role.kubernetes.io/node=
Annotations: flannel.alpha.coreos.com/backend-data: {"VtepMAC":"5a:a7:35:25:88:d4"}
flannel.alpha.coreos.com/backend-type: vxlan
flannel.alpha.coreos.com/kube-subnet-manager: true
flannel.alpha.coreos.com/public-ip: 172.25.27.26
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Tue, 02 Mar 2021 17:47:31 +0000
Taints: <none>
Unschedulable: false
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Tue, 23 Mar 2021 14:38:55 +0000 Tue, 02 Mar 2021 17:47:56 +0000 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Tue, 23 Mar 2021 14:38:55 +0000 Tue, 02 Mar 2021 17:47:56 +0000 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Tue, 23 Mar 2021 14:38:55 +0000 Tue, 02 Mar 2021 17:47:56 +0000 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Tue, 23 Mar 2021 14:38:55 +0000 Tue, 02 Mar 2021 17:49:07 +0000 KubeletReady kubelet is posting ready status
Addresses:
InternalIP: 172.25.27.26
Hostname: 172.25.27.26
Capacity:
cpu: 96
ephemeral-storage: 114356412Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 196496644Ki
pods: 110
Allocatable:
cpu: 95900m
ephemeral-storage: 105390869125
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 196144244Ki
pods: 110
System Info:
Machine ID: 2637027352b94bdfbaf1bb6d06401946
System UUID: 800B863A-10CB-EA11-906E-0017A4403562
Boot ID: 799bbe1a-8015-4250-beb1-c39958381d58
Kernel Version: 3.10.0-957.el7.x86_64
OS Image: CentOS Linux 7 (Core)
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://18.9.9
Kubelet Version: v1.13.12
Kube-Proxy Version: v1.13.12
PodCIDR: 10.234.1.0/24
Non-terminated Pods: (17 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
kube-system kube-flannel-6ntgc 150m (0%) 300m (0%) 64M (0%) 500M (0%) 20d
kube-system kube-proxy-172.25.27.26 150m (0%) 500m (0%) 64M (0%) 2G (0%) 20d
kube-system nginx-proxy-172.25.27.26 25m (0%) 300m (0%) 32M (0%) 512M (0%) 20d
mediakind fluentbit-fluent-bit-8n7vp 0 (0%) 0 (0%) 0 (0%) 0 (0%) 20d
mediakind logrotate-8d4j7 0 (0%) 0 (0%) 0 (0%) 0 (0%) 20d
mediakind metrics-prometheus-node-exporter-q7qqs 0 (0%) 0 (0%) 30Mi (0%) 150Mi (0%) 20d
mediakind mux-bckp-1-stream-processor-mux-65c46756b5-sgprz 0 (0%) 0 (0%) 0 (0%) 0 (0%) 20d
mediakind service-01-enc-hd-encoding-live-worker-6f6c448599-rnx9q 9 (9%) 12 (12%) 1536Mi (0%) 5Gi (2%) 19d
mediakind service-02-iptv-hd-encoding-live-worker-564fd48c7c-lnpk5 9 (9%) 12 (12%) 1536Mi (0%) 5Gi (2%) 6d2h
mediakind service-02-ott-encoding-live-worker-7d74dddc79-d7mn7 4 (4%) 0 (0%) 1G (0%) 0 (0%) 6d17h
mediakind service-03-enc-sd-encoding-live-worker-5d56c8b844-6hsl6 2 (2%) 3 (3%) 500Mi (0%) 2Gi (1%) 19d
mediakind service-05-enc-hd-encoding-live-worker-7bff86bff-nmx4g 9 (9%) 12 (12%) 1536Mi (0%) 5Gi (2%) 19d
mediakind service-08-enc-hd-encoding-live-worker-6cdf78b9c9-r5fbv 9 (9%) 12 (12%) 1536Mi (0%) 5Gi (2%) 8d
mediakind service-dtt01-2-enc-hd-encoding-live-worker-5bdfc9dc9-rfbcc 9 (9%) 12 (12%) 1536Mi (0%) 5Gi (2%) 7d17h
mediakind service-dtt02-3-enc-hd-encoding-live-worker-65d5fb7455-5slpw 9 (9%) 12 (12%) 1536Mi (0%) 5Gi (2%) 7d4h
mediakind service-dtt02-6-enc-sd-encoding-live-worker-7844d9b7d4-2cnz8 2 (2%) 3 (3%) 500Mi (0%) 2Gi (1%) 7d3h
mediakind stream-processor-statmux-bckp-75dc49984-lfl7v 0 (0%) 0 (0%) 0 (0%) 0 (0%) 20d
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 62325m (64%) 79100m (82%)
memory 11903709696 (5%) 39676508416 (19%)
ephemeral-storage 0 (0%) 0 (0%)
Events: <none>

get pods

PODs are groups of containers, each POD can have a single or many containers. Each time a POD is created (from a profile in MDT) the profile details are used to create the POD.
To see what PODs have been deployed, we can use a 'get pods' command.

kubectl -n mediakind get pods
NAME READY STATUS RESTARTS AGE
alarm-7c69d9f46c-tcz8d 3/3 Running 29 78d
alarm-proxy-6f59cbfd6b-ghmdn 1/1 Running 9 78d
alertmanager-metrics-prometheus-operato-alertmanager-0 2/2 Running 18 78d
alertmanager-metrics-prometheus-operato-alertmanager-1 2/2 Running 18 78d
alertmanager-metrics-prometheus-operato-alertmanager-2 2/2 Running 18 78d
automation-config-cf95fd5c7-k6gjm 1/1 Running 29 78d
command-84886544c9-m4gkw 1/1 Running 9 78d
elasticsearch-exporter-5dbf97948c-swlgm 1/1 Running 9 78d
elasticsearch-master-0 1/1 Running 4 34d
elasticsearch-master-1 1/1 Running 4 34d
elasticsearch-master-2 1/1 Running 4 34d
encoding-live-config-65547d7979-t4w7j 1/1 Running 28 78d
encoding-live-ui-connector-74f4458478-bgtvp 1/1 Running 9 78d
esam-proxy-67ffff6d77-8hgxl 1/1 Running 9 78d
etcd-0 1/1 Running 14 78d
etcd-1 1/1 Running 15 78d
etcd-2 1/1 Running 14 78d
failover-api-5d77b57d75-b79xv 3/3 Running 42 78d
fluen-282-kls5n-fluent-bit-2dzvw 1/1 Running 10 78d
fluen-282-kls5n-fluent-bit-2tqzw 1/1 Running 10 78d
fluen-282-kls5n-fluent-bit-4wfv8 1/1 Running 10 78d
fluen-282-kls5n-fluent-bit-6tvrm 1/1 Running 9 78d
fluen-282-kls5n-fluent-bit-fst5t 1/1 Running 9 78d
fluen-282-kls5n-fluent-bit-hxhcv 1/1 Running 9 78d
fluen-282-kls5n-fluent-bit-x5ztw 1/1 Running 10 78d
interface-discovery-cf446c54-6xcmt 1/1 Running 9 78d
kibana-kibana-9bccd7fb8-xmw58 1/1 Running 9 78d
launch-control-749c675669-fkvqm 1/1 Running 9 78d
license-manager-7555998744-qljbm 3/3 Running 23 73d
live-sources-config-665dff566c-9pztm 1/1 Running 28 78d
logrotate-656ts 1/1 Running 9 78d
logrotate-6cgjr 1/1 Running 10 78d
logrotate-fjkpx 1/1 Running 9 78d
logrotate-k47ng 1/1 Running 10 78d
logrotate-kx7dp 1/1 Running 10 78d
logrotate-t6pqg 1/1 Running 9 78d
logrotate-wctw7 1/1 Running 10 78d
metrics-grafana-5df9cd76bd-pxbhp 2/2 Running 18 78d
metrics-kube-state-metrics-556bbc4fbf-vn86v 1/1 Running 9 78d
metrics-prometheus-node-exporter-6zqwd 1/1 Running 10 78d
metrics-prometheus-node-exporter-dm9cf 1/1 Running 10 78d
metrics-prometheus-node-exporter-g8b58 1/1 Running 10 78d
metrics-prometheus-node-exporter-kcnqb 1/1 Running 9 78d
metrics-prometheus-node-exporter-t2pbl 1/1 Running 9 78d
metrics-prometheus-node-exporter-v46j6 1/1 Running 9 78d
metrics-prometheus-node-exporter-vtsqw 1/1 Running 10 78d
metrics-prometheus-operato-operator-5fb8bc499f-8sfsw 2/2 Running 18 78d
mkel-sd1-encoding-live-worker-84596675d7-jbgcp 4/4 Running 0 25m
mkel-sd2-encoding-live-worker-668846bbdd-xtlnn 4/4 Running 0 25m
mongodb-replicaset-0 2/2 Running 18 78d
mongodb-replicaset-1 2/2 Running 18 78d
mongodb-replicaset-2 2/2 Running 18 78d
muconv-config-77f85cf5b9-lm74k 1/1 Running 28 78d
muconv-ui-connector-68dbd6c756-rmxpd 1/1 Running 13 78d
mux-bckp-1-stream-processor-mux-b6c545c89-srpvv 5/5 Running 0 2m19s
mux-main-1-stream-processor-mux-844644dd44-tqqlr 5/5 Running 0 2m50s
nielsen-rtvod-log-archiver-857c6788b5-5swgj 1/1 Running 28 78d
packaging-config-5bd6778cf4-tsjlj 1/1 Running 9 78d
packaging-dvr-auto-577d4b487c-5kw9g 1/1 Running 9 78d
packaging-dynamic-ui-connector-7b65cc476f-jfv8d 1/1 Running 9 78d
packaging-recorder-ui-connector-6496d758c9-ntdjp 1/1 Running 9 78d
placement-api-75dfd78f77-nb74z 1/1 Running 9 78d
prometheus-metrics-prometheus-operato-prometheus-0 3/3 Running 28 78d
prometheus-metrics-prometheus-operato-prometheus-1 3/3 Running 28 78d
prometheus-metrics-prometheus-operato-prometheus-2 3/3 Running 28 78d
rabbitmq-ha-0 1/1 Running 0 27m
rabbitmq-ha-1 1/1 Running 0 26m
rabbitmq-ha-2 1/1 Running 0 26m
redis-5885955657-9rzhf 1/1 Running 9 78d
redis-exporter-prometheus-redis-exporter-b75844d5f-mscxs 1/1 Running 9 78d
server-api-7dbc67f674-qzr76 1/1 Running 10 78d
server-monitor-7fc9bbcd9f-hjz6z 1/1 Running 9 78d
service-api-56d7cfdf7c-qd4tv 2/2 Running 26 78d
snmp-notification-6ddc75cf85-pdlzq 1/1 Running 13 78d
statistic-7cdccbcfcf-qkbb2 1/1 Running 9 78d
stream-processor-config-7578c5bb7b-bgc87 1/1 Running 9 78d
stream-processor-statmux-bckp-6584c7849d-mmzzk 3/3 Running 0 2m50s
stream-processor-statmux-main-9954b49db-9f7s5 3/3 Running 0 3m21s
stream-processor-ui-connector-6bd8c4c9b5-lsdjd 1/1 Running 9 78d
template-api-65c56cfc95-sd49f 1/1 Running 15 78d
unified-ui-cbf7767bd-mqv7t 5/5 Running 51 78d

delete pods

Once the PODs have been listed, it is then possible to delete a POD or PODs. Use the following (this example assumes you have these PODs available for deletion).
Lets say you wish to delete the following three PODS

rabbitmq-ha-0 1/1 Running 0 27m
rabbitmq-ha-1 1/1 Running 0 26m
rabbitmq-ha-2 1/1 Running 0 26m

You would use the following command: (assuming you are using the namespace mediakind).

kubectl -n mediakind delete pod rabbitmq-ha-0 rabbitmq-ha-1 rabbitmq-ha-2
pod "rabbitmq-ha-0" deleted
pod "rabbitmq-ha-1" deleted
pod "rabbitmq-ha-2" deleted

Kubernetes would the start to rebuild these PODs if they are still currently deployed in IPHE.

get deploy

'get deploy' is similar to get PODs, but it shows the deployed components of your system rather than all of the system, so you don't see Rabbit, Mongo etc, but you do see what you want in terms of services and their components

kubectl -n mediakind get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
alarm 1/1 1 1 78d
alarm-proxy 1/1 1 1 78d
automation-config 1/1 1 1 78d
command 1/1 1 1 78d
elasticsearch-exporter 1/1 1 1 78d
encoding-live-config 1/1 1 1 78d
encoding-live-ui-connector 1/1 1 1 78d
esam-proxy 1/1 1 1 78d
failover-api 1/1 1 1 78d
flow 0/0 0 0 78d
interface-discovery 1/1 1 1 78d
kibana-kibana 1/1 1 1 78d
launch-control 1/1 1 1 78d
license-manager 1/1 1 1 76d
live-sources-config 1/1 1 1 78d
metrics-grafana 1/1 1 1 78d
metrics-kube-state-metrics 1/1 1 1 78d
metrics-prometheus-operato-operator 1/1 1 1 78d
mfvp-probe-light 0/0 0 0 78d
mkel-sd1-encoding-live-worker 1/1 1 1 46m
mkel-sd2-encoding-live-worker 1/1 1 1 46m
muconv-config 1/1 1 1 78d
muconv-ui-connector 1/1 1 1 78d
mux-bckp-1-stream-processor-mux 1/1 1 1 23m
mux-main-1-stream-processor-mux 1/1 1 1 24m
nielsen-rtvod-log-archiver 1/1 1 1 78d
packaging-config 1/1 1 1 78d
packaging-dvr-auto 1/1 1 1 78d
packaging-dynamic-ui-connector 1/1 1 1 78d
packaging-recorder-ui-connector 1/1 1 1 78d
placement-api 1/1 1 1 78d
redis 1/1 1 1 78d
redis-exporter-prometheus-redis-exporter 1/1 1 1 78d
server-api 1/1 1 1 78d
server-monitor 1/1 1 1 78d
service-api 1/1 1 1 78d
snmp-notification 1/1 1 1 78d
statistic 1/1 1 1 78d
stream-processor-asi 0/0 0 0 24m
stream-processor-config 1/1 1 1 78d
stream-processor-statmux-bckp 1/1 1 1 24m
stream-processor-statmux-main 1/1 1 1 24m
stream-processor-ui-connector 1/1 1 1 78d
template-api 1/1 1 1 78d
unified-ui 1/1 1 1 78d

get statefulset

Stateful set is for pods that have to maintain state, say even after a reboot, and tries to put things back in the same place as before, where as Deployments dont care where they put replicasets/pods.

kubectl -n mediakind get statefulset
NAME READY AGE
alertmanager-metrics-prometheus-operato-alertmanager 3/3 78d
elasticsearch-master 3/3 78d
etcd 3/3 78d
mongodb-replicaset 3/3 78d
prometheus-metrics-prometheus-operato-prometheus 3/3 78d
rabbitmq-ha