This is an old revision of the document!
Table of Contents
Kubernetes K8
Apr 2021
Overview
Containers contain functions for software components, a container can be a web server, a data processor or an data I/O device, anything really that you used to be able to run on a bare metal server or a VM. Containers can run on multiple servers in multiple locations.
Kubernetes provides an open-source API that controls how and where those containers will run, and manages continer failure and restarts.
Kubernetes Commands
There are many Kubernets commands, here we will explore some that will be useful when dealing with MediaKind HeadEnd solutions (IPHE).
get ns
kubectl -n mediakind get nodes
PODs run in a particular namespace, there can be more than one namespace in a system. To be able to use any commands you have to know which namespace you wish to use, and specify that namespace in your command line.
Use the following command to show available namespace.
kubectl get ns NAME STATUS AGE default Active 78d kube-public Active 78d kube-system Active 78d mediakind Active 78d
get nodes
kubectl -n mediakind get nodes
In the above example, kubectl is the command, mediakind is the namespace we are dealing with and get nodes is the parameters for the command.
NAME STATUS ROLES AGE VERSION 172.25.27.17 Ready master,node 20d v1.13.12 172.25.27.20 Ready master,node 20d v1.13.12 172.25.27.21 Ready master,node 20d v1.13.12 172.25.27.26 Ready node 20d v1.13.12
kubectl -n mediakind get nodes returns the number of 'servers' in the deployment, these nodes can be bare metal or Virtual Machines.
describe node
kubectl -n mediakind describe node ipaddress
Above the command is describing a particular node in the system, the node IP address was derived by using the get node command.
kubectl -n mediakind describe node 172.25.27.26 Name: 172.25.27.26 Roles: node Labels: beta.kubernetes.io/arch=amd64 beta.kubernetes.io/os=linux encoding-hd-services=enabled encoding-ott-services=enabled encoding-sd-services=enabled kubernetes.io/hostname=172.25.27.26 mux-bckp=enabled node-role.kubernetes.io/node= Annotations: flannel.alpha.coreos.com/backend-data: {"VtepMAC":"5a:a7:35:25:88:d4"} flannel.alpha.coreos.com/backend-type: vxlan flannel.alpha.coreos.com/kube-subnet-manager: true flannel.alpha.coreos.com/public-ip: 172.25.27.26 node.alpha.kubernetes.io/ttl: 0 volumes.kubernetes.io/controller-managed-attach-detach: true CreationTimestamp: Tue, 02 Mar 2021 17:47:31 +0000 Taints: <none> Unschedulable: false Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- MemoryPressure False Tue, 23 Mar 2021 14:38:55 +0000 Tue, 02 Mar 2021 17:47:56 +0000 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Tue, 23 Mar 2021 14:38:55 +0000 Tue, 02 Mar 2021 17:47:56 +0000 KubeletHasNoDiskPressure kubelet has no disk pressure PIDPressure False Tue, 23 Mar 2021 14:38:55 +0000 Tue, 02 Mar 2021 17:47:56 +0000 KubeletHasSufficientPID kubelet has sufficient PID available Ready True Tue, 23 Mar 2021 14:38:55 +0000 Tue, 02 Mar 2021 17:49:07 +0000 KubeletReady kubelet is posting ready status Addresses: InternalIP: 172.25.27.26 Hostname: 172.25.27.26 Capacity: cpu: 96 ephemeral-storage: 114356412Ki hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 196496644Ki pods: 110 Allocatable: cpu: 95900m ephemeral-storage: 105390869125 hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 196144244Ki pods: 110 System Info: Machine ID: 2637027352b94bdfbaf1bb6d06401946 System UUID: 800B863A-10CB-EA11-906E-0017A4403562 Boot ID: 799bbe1a-8015-4250-beb1-c39958381d58 Kernel Version: 3.10.0-957.el7.x86_64 OS Image: CentOS Linux 7 (Core) Operating System: linux Architecture: amd64 Container Runtime Version: docker://18.9.9 Kubelet Version: v1.13.12 Kube-Proxy Version: v1.13.12 PodCIDR: 10.234.1.0/24 Non-terminated Pods: (17 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE --------- ---- ------------ ---------- --------------- ------------- --- kube-system kube-flannel-6ntgc 150m (0%) 300m (0%) 64M (0%) 500M (0%) 20d kube-system kube-proxy-172.25.27.26 150m (0%) 500m (0%) 64M (0%) 2G (0%) 20d kube-system nginx-proxy-172.25.27.26 25m (0%) 300m (0%) 32M (0%) 512M (0%) 20d mediakind fluentbit-fluent-bit-8n7vp 0 (0%) 0 (0%) 0 (0%) 0 (0%) 20d mediakind logrotate-8d4j7 0 (0%) 0 (0%) 0 (0%) 0 (0%) 20d mediakind metrics-prometheus-node-exporter-q7qqs 0 (0%) 0 (0%) 30Mi (0%) 150Mi (0%) 20d mediakind mux-bckp-1-stream-processor-mux-65c46756b5-sgprz 0 (0%) 0 (0%) 0 (0%) 0 (0%) 20d mediakind service-01-enc-hd-encoding-live-worker-6f6c448599-rnx9q 9 (9%) 12 (12%) 1536Mi (0%) 5Gi (2%) 19d mediakind service-02-iptv-hd-encoding-live-worker-564fd48c7c-lnpk5 9 (9%) 12 (12%) 1536Mi (0%) 5Gi (2%) 6d2h mediakind service-02-ott-encoding-live-worker-7d74dddc79-d7mn7 4 (4%) 0 (0%) 1G (0%) 0 (0%) 6d17h mediakind service-03-enc-sd-encoding-live-worker-5d56c8b844-6hsl6 2 (2%) 3 (3%) 500Mi (0%) 2Gi (1%) 19d mediakind service-05-enc-hd-encoding-live-worker-7bff86bff-nmx4g 9 (9%) 12 (12%) 1536Mi (0%) 5Gi (2%) 19d mediakind service-08-enc-hd-encoding-live-worker-6cdf78b9c9-r5fbv 9 (9%) 12 (12%) 1536Mi (0%) 5Gi (2%) 8d mediakind service-dtt01-2-enc-hd-encoding-live-worker-5bdfc9dc9-rfbcc 9 (9%) 12 (12%) 1536Mi (0%) 5Gi (2%) 7d17h mediakind service-dtt02-3-enc-hd-encoding-live-worker-65d5fb7455-5slpw 9 (9%) 12 (12%) 1536Mi (0%) 5Gi (2%) 7d4h mediakind service-dtt02-6-enc-sd-encoding-live-worker-7844d9b7d4-2cnz8 2 (2%) 3 (3%) 500Mi (0%) 2Gi (1%) 7d3h mediakind stream-processor-statmux-bckp-75dc49984-lfl7v 0 (0%) 0 (0%) 0 (0%) 0 (0%) 20d Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits -------- -------- ------ cpu 62325m (64%) 79100m (82%) memory 11903709696 (5%) 39676508416 (19%) ephemeral-storage 0 (0%) 0 (0%) Events: <none>
get pods
PODs are groups of containers, each POD can have a single or many containers. Each time a POD is created (from a profile in MDT) the profile details are used to create the POD.
To see what PODs have been deployed, we can use a 'get pods' command.
kubectl -n mediakind get pods NAME READY STATUS RESTARTS AGE alarm-7c69d9f46c-tcz8d 3/3 Running 29 78d alarm-proxy-6f59cbfd6b-ghmdn 1/1 Running 9 78d alertmanager-metrics-prometheus-operato-alertmanager-0 2/2 Running 18 78d alertmanager-metrics-prometheus-operato-alertmanager-1 2/2 Running 18 78d alertmanager-metrics-prometheus-operato-alertmanager-2 2/2 Running 18 78d automation-config-cf95fd5c7-k6gjm 1/1 Running 29 78d command-84886544c9-m4gkw 1/1 Running 9 78d elasticsearch-exporter-5dbf97948c-swlgm 1/1 Running 9 78d elasticsearch-master-0 1/1 Running 4 34d elasticsearch-master-1 1/1 Running 4 34d elasticsearch-master-2 1/1 Running 4 34d encoding-live-config-65547d7979-t4w7j 1/1 Running 28 78d encoding-live-ui-connector-74f4458478-bgtvp 1/1 Running 9 78d esam-proxy-67ffff6d77-8hgxl 1/1 Running 9 78d etcd-0 1/1 Running 14 78d etcd-1 1/1 Running 15 78d etcd-2 1/1 Running 14 78d failover-api-5d77b57d75-b79xv 3/3 Running 42 78d fluen-282-kls5n-fluent-bit-2dzvw 1/1 Running 10 78d fluen-282-kls5n-fluent-bit-2tqzw 1/1 Running 10 78d fluen-282-kls5n-fluent-bit-4wfv8 1/1 Running 10 78d fluen-282-kls5n-fluent-bit-6tvrm 1/1 Running 9 78d fluen-282-kls5n-fluent-bit-fst5t 1/1 Running 9 78d fluen-282-kls5n-fluent-bit-hxhcv 1/1 Running 9 78d fluen-282-kls5n-fluent-bit-x5ztw 1/1 Running 10 78d interface-discovery-cf446c54-6xcmt 1/1 Running 9 78d kibana-kibana-9bccd7fb8-xmw58 1/1 Running 9 78d launch-control-749c675669-fkvqm 1/1 Running 9 78d license-manager-7555998744-qljbm 3/3 Running 23 73d live-sources-config-665dff566c-9pztm 1/1 Running 28 78d logrotate-656ts 1/1 Running 9 78d logrotate-6cgjr 1/1 Running 10 78d logrotate-fjkpx 1/1 Running 9 78d logrotate-k47ng 1/1 Running 10 78d logrotate-kx7dp 1/1 Running 10 78d logrotate-t6pqg 1/1 Running 9 78d logrotate-wctw7 1/1 Running 10 78d metrics-grafana-5df9cd76bd-pxbhp 2/2 Running 18 78d metrics-kube-state-metrics-556bbc4fbf-vn86v 1/1 Running 9 78d metrics-prometheus-node-exporter-6zqwd 1/1 Running 10 78d metrics-prometheus-node-exporter-dm9cf 1/1 Running 10 78d metrics-prometheus-node-exporter-g8b58 1/1 Running 10 78d metrics-prometheus-node-exporter-kcnqb 1/1 Running 9 78d metrics-prometheus-node-exporter-t2pbl 1/1 Running 9 78d metrics-prometheus-node-exporter-v46j6 1/1 Running 9 78d metrics-prometheus-node-exporter-vtsqw 1/1 Running 10 78d metrics-prometheus-operato-operator-5fb8bc499f-8sfsw 2/2 Running 18 78d mkel-sd1-encoding-live-worker-84596675d7-jbgcp 4/4 Running 0 25m mkel-sd2-encoding-live-worker-668846bbdd-xtlnn 4/4 Running 0 25m mongodb-replicaset-0 2/2 Running 18 78d mongodb-replicaset-1 2/2 Running 18 78d mongodb-replicaset-2 2/2 Running 18 78d muconv-config-77f85cf5b9-lm74k 1/1 Running 28 78d muconv-ui-connector-68dbd6c756-rmxpd 1/1 Running 13 78d mux-bckp-1-stream-processor-mux-b6c545c89-srpvv 5/5 Running 0 2m19s mux-main-1-stream-processor-mux-844644dd44-tqqlr 5/5 Running 0 2m50s nielsen-rtvod-log-archiver-857c6788b5-5swgj 1/1 Running 28 78d packaging-config-5bd6778cf4-tsjlj 1/1 Running 9 78d packaging-dvr-auto-577d4b487c-5kw9g 1/1 Running 9 78d packaging-dynamic-ui-connector-7b65cc476f-jfv8d 1/1 Running 9 78d packaging-recorder-ui-connector-6496d758c9-ntdjp 1/1 Running 9 78d placement-api-75dfd78f77-nb74z 1/1 Running 9 78d prometheus-metrics-prometheus-operato-prometheus-0 3/3 Running 28 78d prometheus-metrics-prometheus-operato-prometheus-1 3/3 Running 28 78d prometheus-metrics-prometheus-operato-prometheus-2 3/3 Running 28 78d rabbitmq-ha-0 1/1 Running 0 27m rabbitmq-ha-1 1/1 Running 0 26m rabbitmq-ha-2 1/1 Running 0 26m redis-5885955657-9rzhf 1/1 Running 9 78d redis-exporter-prometheus-redis-exporter-b75844d5f-mscxs 1/1 Running 9 78d server-api-7dbc67f674-qzr76 1/1 Running 10 78d server-monitor-7fc9bbcd9f-hjz6z 1/1 Running 9 78d service-api-56d7cfdf7c-qd4tv 2/2 Running 26 78d snmp-notification-6ddc75cf85-pdlzq 1/1 Running 13 78d statistic-7cdccbcfcf-qkbb2 1/1 Running 9 78d stream-processor-config-7578c5bb7b-bgc87 1/1 Running 9 78d stream-processor-statmux-bckp-6584c7849d-mmzzk 3/3 Running 0 2m50s stream-processor-statmux-main-9954b49db-9f7s5 3/3 Running 0 3m21s stream-processor-ui-connector-6bd8c4c9b5-lsdjd 1/1 Running 9 78d template-api-65c56cfc95-sd49f 1/1 Running 15 78d unified-ui-cbf7767bd-mqv7t 5/5 Running 51 78d
delete pods
Once the PODs have been listed, it is then possible to delete a POD or PODs. Use the following (this example assumes you have these PODs available for deletion).
Lets say you wish to delete the following three PODS
rabbitmq-ha-0 1/1 Running 0 27m rabbitmq-ha-1 1/1 Running 0 26m rabbitmq-ha-2 1/1 Running 0 26m
You would use the following command: (assuming you are using the namespace mediakind).
kubectl -n mediakind delete pod rabbitmq-ha-0 rabbitmq-ha-1 rabbitmq-ha-2 pod "rabbitmq-ha-0" deleted pod "rabbitmq-ha-1" deleted pod "rabbitmq-ha-2" deleted
Kubernetes would the start to rebuild these PODs if they are still currently deployed in IPHE.
get deploy
'get deploy' is similar to get PODs, but it shows the deployed components of your system rather than all of the system, so you don't see Rabbit, Mongo etc, but you do see what you want in terms of services and their components
kubectl -n mediakind get deploy NAME READY UP-TO-DATE AVAILABLE AGE alarm 1/1 1 1 78d alarm-proxy 1/1 1 1 78d automation-config 1/1 1 1 78d command 1/1 1 1 78d elasticsearch-exporter 1/1 1 1 78d encoding-live-config 1/1 1 1 78d encoding-live-ui-connector 1/1 1 1 78d esam-proxy 1/1 1 1 78d failover-api 1/1 1 1 78d flow 0/0 0 0 78d interface-discovery 1/1 1 1 78d kibana-kibana 1/1 1 1 78d launch-control 1/1 1 1 78d license-manager 1/1 1 1 76d live-sources-config 1/1 1 1 78d metrics-grafana 1/1 1 1 78d metrics-kube-state-metrics 1/1 1 1 78d metrics-prometheus-operato-operator 1/1 1 1 78d mfvp-probe-light 0/0 0 0 78d mkel-sd1-encoding-live-worker 1/1 1 1 46m mkel-sd2-encoding-live-worker 1/1 1 1 46m muconv-config 1/1 1 1 78d muconv-ui-connector 1/1 1 1 78d mux-bckp-1-stream-processor-mux 1/1 1 1 23m mux-main-1-stream-processor-mux 1/1 1 1 24m nielsen-rtvod-log-archiver 1/1 1 1 78d packaging-config 1/1 1 1 78d packaging-dvr-auto 1/1 1 1 78d packaging-dynamic-ui-connector 1/1 1 1 78d packaging-recorder-ui-connector 1/1 1 1 78d placement-api 1/1 1 1 78d redis 1/1 1 1 78d redis-exporter-prometheus-redis-exporter 1/1 1 1 78d server-api 1/1 1 1 78d server-monitor 1/1 1 1 78d service-api 1/1 1 1 78d snmp-notification 1/1 1 1 78d statistic 1/1 1 1 78d stream-processor-asi 0/0 0 0 24m stream-processor-config 1/1 1 1 78d stream-processor-statmux-bckp 1/1 1 1 24m stream-processor-statmux-main 1/1 1 1 24m stream-processor-ui-connector 1/1 1 1 78d template-api 1/1 1 1 78d unified-ui 1/1 1 1 78d
get statefulset
Stateful set is for pods that have to maintain state, say even after a reboot, and tries to put things back in the same place as before, where as Deployments dont care where they put replicasets/pods.
kubectl -n mediakind get statefulset NAME READY AGE alertmanager-metrics-prometheus-operato-alertmanager 3/3 78d elasticsearch-master 3/3 78d etcd 3/3 78d mongodb-replicaset 3/3 78d prometheus-metrics-prometheus-operato-prometheus 3/3 78d rabbitmq-ha