kubernetes_k8
This is an old revision of the document!
Table of Contents
Kubernetes K8
Apr 2021
Overview
Containers contain functions for software components, a container can be a web server, a data processor or an data I/O device, anything really that you used to be able to run on a bare metal server or a VM. Containers can run on multiple servers in multiple locations.
Kubernetes provides an open-source API that controls how and where those containers will run, and manages continer failure and restarts.
Kubernetes Commands
There are many Kubernets commands, here we will explore some that will be useful when dealing with MediaKind HeadEnd solutions (IPHE).
get ns
kubectl -n mediakind get nodes
PODs run in a particular namespace, there can be more than one namespace in a system. To be able to use any commands you have to know which namespace you wish to use, and specify that namespace in your command line.
Use the following command to show available namespace.
kubectl get ns NAME STATUS AGE default Active 78d kube-public Active 78d kube-system Active 78d mediakind Active 78d
get nodes
kubectl -n mediakind get nodes
In the above example, kubectl is the command, mediakind is the namespace we are dealing with and get nodes is the parameters for the command.
NAME STATUS ROLES AGE VERSION 172.25.27.17 Ready master,node 20d v1.13.12 172.25.27.20 Ready master,node 20d v1.13.12 172.25.27.21 Ready master,node 20d v1.13.12 172.25.27.26 Ready node 20d v1.13.12
kubectl -n mediakind get nodes returns the number of 'servers' in the deployment, these nodes can be bare metal or Virtual Machines.
describe node
kubectl -n mediakind describe node ipaddress
Above the command is describing a particular node in the system, the node IP address was derived by using the get node command.
kubectl -n mediakind describe node 172.25.27.26
Name: 172.25.27.26
Roles: node
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
encoding-hd-services=enabled
encoding-ott-services=enabled
encoding-sd-services=enabled
kubernetes.io/hostname=172.25.27.26
mux-bckp=enabled
node-role.kubernetes.io/node=
Annotations: flannel.alpha.coreos.com/backend-data: {"VtepMAC":"5a:a7:35:25:88:d4"}
flannel.alpha.coreos.com/backend-type: vxlan
flannel.alpha.coreos.com/kube-subnet-manager: true
flannel.alpha.coreos.com/public-ip: 172.25.27.26
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Tue, 02 Mar 2021 17:47:31 +0000
Taints: <none>
Unschedulable: false
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Tue, 23 Mar 2021 14:38:55 +0000 Tue, 02 Mar 2021 17:47:56 +0000 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Tue, 23 Mar 2021 14:38:55 +0000 Tue, 02 Mar 2021 17:47:56 +0000 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Tue, 23 Mar 2021 14:38:55 +0000 Tue, 02 Mar 2021 17:47:56 +0000 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Tue, 23 Mar 2021 14:38:55 +0000 Tue, 02 Mar 2021 17:49:07 +0000 KubeletReady kubelet is posting ready status
Addresses:
InternalIP: 172.25.27.26
Hostname: 172.25.27.26
Capacity:
cpu: 96
ephemeral-storage: 114356412Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 196496644Ki
pods: 110
Allocatable:
cpu: 95900m
ephemeral-storage: 105390869125
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 196144244Ki
pods: 110
System Info:
Machine ID: 2637027352b94bdfbaf1bb6d06401946
System UUID: 800B863A-10CB-EA11-906E-0017A4403562
Boot ID: 799bbe1a-8015-4250-beb1-c39958381d58
Kernel Version: 3.10.0-957.el7.x86_64
OS Image: CentOS Linux 7 (Core)
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://18.9.9
Kubelet Version: v1.13.12
Kube-Proxy Version: v1.13.12
PodCIDR: 10.234.1.0/24
Non-terminated Pods: (17 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
kube-system kube-flannel-6ntgc 150m (0%) 300m (0%) 64M (0%) 500M (0%) 20d
kube-system kube-proxy-172.25.27.26 150m (0%) 500m (0%) 64M (0%) 2G (0%) 20d
kube-system nginx-proxy-172.25.27.26 25m (0%) 300m (0%) 32M (0%) 512M (0%) 20d
mediakind fluentbit-fluent-bit-8n7vp 0 (0%) 0 (0%) 0 (0%) 0 (0%) 20d
mediakind logrotate-8d4j7 0 (0%) 0 (0%) 0 (0%) 0 (0%) 20d
mediakind metrics-prometheus-node-exporter-q7qqs 0 (0%) 0 (0%) 30Mi (0%) 150Mi (0%) 20d
mediakind mux-bckp-1-stream-processor-mux-65c46756b5-sgprz 0 (0%) 0 (0%) 0 (0%) 0 (0%) 20d
mediakind service-01-enc-hd-encoding-live-worker-6f6c448599-rnx9q 9 (9%) 12 (12%) 1536Mi (0%) 5Gi (2%) 19d
mediakind service-02-iptv-hd-encoding-live-worker-564fd48c7c-lnpk5 9 (9%) 12 (12%) 1536Mi (0%) 5Gi (2%) 6d2h
mediakind service-02-ott-encoding-live-worker-7d74dddc79-d7mn7 4 (4%) 0 (0%) 1G (0%) 0 (0%) 6d17h
mediakind service-03-enc-sd-encoding-live-worker-5d56c8b844-6hsl6 2 (2%) 3 (3%) 500Mi (0%) 2Gi (1%) 19d
mediakind service-05-enc-hd-encoding-live-worker-7bff86bff-nmx4g 9 (9%) 12 (12%) 1536Mi (0%) 5Gi (2%) 19d
mediakind service-08-enc-hd-encoding-live-worker-6cdf78b9c9-r5fbv 9 (9%) 12 (12%) 1536Mi (0%) 5Gi (2%) 8d
mediakind service-dtt01-2-enc-hd-encoding-live-worker-5bdfc9dc9-rfbcc 9 (9%) 12 (12%) 1536Mi (0%) 5Gi (2%) 7d17h
mediakind service-dtt02-3-enc-hd-encoding-live-worker-65d5fb7455-5slpw 9 (9%) 12 (12%) 1536Mi (0%) 5Gi (2%) 7d4h
mediakind service-dtt02-6-enc-sd-encoding-live-worker-7844d9b7d4-2cnz8 2 (2%) 3 (3%) 500Mi (0%) 2Gi (1%) 7d3h
mediakind stream-processor-statmux-bckp-75dc49984-lfl7v 0 (0%) 0 (0%) 0 (0%) 0 (0%) 20d
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 62325m (64%) 79100m (82%)
memory 11903709696 (5%) 39676508416 (19%)
ephemeral-storage 0 (0%) 0 (0%)
Events: <none>
get pods
PODs are groups of containers, each POD can have a single or many containers. Each time a POD is created (from a profile in MDT) the profile details are used to create the POD.
To see what PODs have been deployed, we can use a 'get pods' command.
kubectl -n mediakind get pods NAME READY STATUS RESTARTS AGE alarm-7c69d9f46c-tcz8d 3/3 Running 29 78d alarm-proxy-6f59cbfd6b-ghmdn 1/1 Running 9 78d alertmanager-metrics-prometheus-operato-alertmanager-0 2/2 Running 18 78d alertmanager-metrics-prometheus-operato-alertmanager-1 2/2 Running 18 78d alertmanager-metrics-prometheus-operato-alertmanager-2 2/2 Running 18 78d automation-config-cf95fd5c7-k6gjm 1/1 Running 29 78d command-84886544c9-m4gkw 1/1 Running 9 78d elasticsearch-exporter-5dbf97948c-swlgm 1/1 Running 9 78d elasticsearch-master-0 1/1 Running 4 34d elasticsearch-master-1 1/1 Running 4 34d elasticsearch-master-2 1/1 Running 4 34d encoding-live-config-65547d7979-t4w7j 1/1 Running 28 78d encoding-live-ui-connector-74f4458478-bgtvp 1/1 Running 9 78d esam-proxy-67ffff6d77-8hgxl 1/1 Running 9 78d etcd-0 1/1 Running 14 78d etcd-1 1/1 Running 15 78d etcd-2 1/1 Running 14 78d failover-api-5d77b57d75-b79xv 3/3 Running 42 78d fluen-282-kls5n-fluent-bit-2dzvw 1/1 Running 10 78d fluen-282-kls5n-fluent-bit-2tqzw 1/1 Running 10 78d fluen-282-kls5n-fluent-bit-4wfv8 1/1 Running 10 78d fluen-282-kls5n-fluent-bit-6tvrm 1/1 Running 9 78d fluen-282-kls5n-fluent-bit-fst5t 1/1 Running 9 78d fluen-282-kls5n-fluent-bit-hxhcv 1/1 Running 9 78d fluen-282-kls5n-fluent-bit-x5ztw 1/1 Running 10 78d interface-discovery-cf446c54-6xcmt 1/1 Running 9 78d kibana-kibana-9bccd7fb8-xmw58 1/1 Running 9 78d launch-control-749c675669-fkvqm 1/1 Running 9 78d license-manager-7555998744-qljbm 3/3 Running 23 73d live-sources-config-665dff566c-9pztm 1/1 Running 28 78d logrotate-656ts 1/1 Running 9 78d logrotate-6cgjr 1/1 Running 10 78d logrotate-fjkpx 1/1 Running 9 78d logrotate-k47ng 1/1 Running 10 78d logrotate-kx7dp 1/1 Running 10 78d logrotate-t6pqg 1/1 Running 9 78d logrotate-wctw7 1/1 Running 10 78d metrics-grafana-5df9cd76bd-pxbhp 2/2 Running 18 78d metrics-kube-state-metrics-556bbc4fbf-vn86v 1/1 Running 9 78d metrics-prometheus-node-exporter-6zqwd 1/1 Running 10 78d metrics-prometheus-node-exporter-dm9cf 1/1 Running 10 78d metrics-prometheus-node-exporter-g8b58 1/1 Running 10 78d metrics-prometheus-node-exporter-kcnqb 1/1 Running 9 78d metrics-prometheus-node-exporter-t2pbl 1/1 Running 9 78d metrics-prometheus-node-exporter-v46j6 1/1 Running 9 78d metrics-prometheus-node-exporter-vtsqw 1/1 Running 10 78d metrics-prometheus-operato-operator-5fb8bc499f-8sfsw 2/2 Running 18 78d mkel-sd1-encoding-live-worker-84596675d7-jbgcp 4/4 Running 0 25m mkel-sd2-encoding-live-worker-668846bbdd-xtlnn 4/4 Running 0 25m mongodb-replicaset-0 2/2 Running 18 78d mongodb-replicaset-1 2/2 Running 18 78d mongodb-replicaset-2 2/2 Running 18 78d muconv-config-77f85cf5b9-lm74k 1/1 Running 28 78d muconv-ui-connector-68dbd6c756-rmxpd 1/1 Running 13 78d mux-bckp-1-stream-processor-mux-b6c545c89-srpvv 5/5 Running 0 2m19s mux-main-1-stream-processor-mux-844644dd44-tqqlr 5/5 Running 0 2m50s nielsen-rtvod-log-archiver-857c6788b5-5swgj 1/1 Running 28 78d packaging-config-5bd6778cf4-tsjlj 1/1 Running 9 78d packaging-dvr-auto-577d4b487c-5kw9g 1/1 Running 9 78d packaging-dynamic-ui-connector-7b65cc476f-jfv8d 1/1 Running 9 78d packaging-recorder-ui-connector-6496d758c9-ntdjp 1/1 Running 9 78d placement-api-75dfd78f77-nb74z 1/1 Running 9 78d prometheus-metrics-prometheus-operato-prometheus-0 3/3 Running 28 78d prometheus-metrics-prometheus-operato-prometheus-1 3/3 Running 28 78d prometheus-metrics-prometheus-operato-prometheus-2 3/3 Running 28 78d rabbitmq-ha-0 1/1 Running 0 27m rabbitmq-ha-1 1/1 Running 0 26m rabbitmq-ha-2 1/1 Running 0 26m redis-5885955657-9rzhf 1/1 Running 9 78d redis-exporter-prometheus-redis-exporter-b75844d5f-mscxs 1/1 Running 9 78d server-api-7dbc67f674-qzr76 1/1 Running 10 78d server-monitor-7fc9bbcd9f-hjz6z 1/1 Running 9 78d service-api-56d7cfdf7c-qd4tv 2/2 Running 26 78d snmp-notification-6ddc75cf85-pdlzq 1/1 Running 13 78d statistic-7cdccbcfcf-qkbb2 1/1 Running 9 78d stream-processor-config-7578c5bb7b-bgc87 1/1 Running 9 78d stream-processor-statmux-bckp-6584c7849d-mmzzk 3/3 Running 0 2m50s stream-processor-statmux-main-9954b49db-9f7s5 3/3 Running 0 3m21s stream-processor-ui-connector-6bd8c4c9b5-lsdjd 1/1 Running 9 78d template-api-65c56cfc95-sd49f 1/1 Running 15 78d unified-ui-cbf7767bd-mqv7t 5/5 Running 51 78d
delete pods
Once the PODs have been listed, it is then possible to delete a POD or PODs. Use the following (this example assumes you have these PODs available for deletion).
Lets say you wish to delete the following three PODS
rabbitmq-ha-0 1/1 Running 0 27m rabbitmq-ha-1 1/1 Running 0 26m rabbitmq-ha-2 1/1 Running 0 26m
You would use the following command: (assuming you are using the namespace mediakind).
kubectl -n mediakind delete pod rabbitmq-ha-0 rabbitmq-ha-1 rabbitmq-ha-2 pod "rabbitmq-ha-0" deleted pod "rabbitmq-ha-1" deleted pod "rabbitmq-ha-2" deleted
Kubernetes would the start to rebuild these PODs if they are still currently deployed in IPHE.
get deploy
'get deploy' is similar to get PODs, but it shows the deployed components of your system rather than all of the system, so you don't see Rabbit, Mongo etc, but you do see what you want in terms of services and their components
kubectl -n mediakind get deploy NAME READY UP-TO-DATE AVAILABLE AGE alarm 1/1 1 1 78d alarm-proxy 1/1 1 1 78d automation-config 1/1 1 1 78d command 1/1 1 1 78d elasticsearch-exporter 1/1 1 1 78d encoding-live-config 1/1 1 1 78d encoding-live-ui-connector 1/1 1 1 78d esam-proxy 1/1 1 1 78d failover-api 1/1 1 1 78d flow 0/0 0 0 78d interface-discovery 1/1 1 1 78d kibana-kibana 1/1 1 1 78d launch-control 1/1 1 1 78d license-manager 1/1 1 1 76d live-sources-config 1/1 1 1 78d metrics-grafana 1/1 1 1 78d metrics-kube-state-metrics 1/1 1 1 78d metrics-prometheus-operato-operator 1/1 1 1 78d mfvp-probe-light 0/0 0 0 78d mkel-sd1-encoding-live-worker 1/1 1 1 46m mkel-sd2-encoding-live-worker 1/1 1 1 46m muconv-config 1/1 1 1 78d muconv-ui-connector 1/1 1 1 78d mux-bckp-1-stream-processor-mux 1/1 1 1 23m mux-main-1-stream-processor-mux 1/1 1 1 24m nielsen-rtvod-log-archiver 1/1 1 1 78d packaging-config 1/1 1 1 78d packaging-dvr-auto 1/1 1 1 78d packaging-dynamic-ui-connector 1/1 1 1 78d packaging-recorder-ui-connector 1/1 1 1 78d placement-api 1/1 1 1 78d redis 1/1 1 1 78d redis-exporter-prometheus-redis-exporter 1/1 1 1 78d server-api 1/1 1 1 78d server-monitor 1/1 1 1 78d service-api 1/1 1 1 78d snmp-notification 1/1 1 1 78d statistic 1/1 1 1 78d stream-processor-asi 0/0 0 0 24m stream-processor-config 1/1 1 1 78d stream-processor-statmux-bckp 1/1 1 1 24m stream-processor-statmux-main 1/1 1 1 24m stream-processor-ui-connector 1/1 1 1 78d template-api 1/1 1 1 78d unified-ui 1/1 1 1 78d
get statefulset
Stateful set is for pods that have to maintain state, say even after a reboot, and tries to put PODs back in the same place (server or instance) as before, where as Deployments dont care where they put replicasets/pods.
kubectl -n mediakind get statefulset NAME READY AGE alertmanager-metrics-prometheus-operato-alertmanager 3/3 78d elasticsearch-master 3/3 78d etcd 3/3 78d mongodb-replicaset 3/3 78d prometheus-metrics-prometheus-operato-prometheus 3/3 78d rabbitmq-ha
get worker pods
Getting the worker pods provides a list of running services that should match the list of services that you created in Controller. This is the service name followed by the POD name.
kubectl -n mediakind get pods -o wide |grep "worker" service-01-enc-hd-encoding-live-worker-6f6c448599-rnx9q 4/4 Running 9 21d 172.25.27.26 172.25.27.26 <none> <none> service-01-enc-sd-encoding-live-worker-bc678ff64-7hq2s 4/4 Running 20 21d 172.25.27.20 172.25.27.20 <none> <none> service-01-iptv-hd-encoding-live-worker-74f9546cb7-n44xp 4/4 Running 0 8d 172.25.27.20 172.25.27.20 <none> <none> service-01-ott-encoding-live-worker-75d78f546f-rk7wd 4/4 Running 0 9d 172.25.27.21 172.25.27.21 <none> <none> service-02-enc-hd-encoding-live-worker-7f7ffd7b59-bhxwx 4/4 Running 3 15d 172.25.27.20 172.25.27.20 <none> <none> service-02-enc-sd-encoding-live-worker-648d9fccfb-xmr2v 4/4 Running 3 15d 172.25.27.17 172.25.27.17 <none> <none> service-02-iptv-hd-encoding-live-worker-564fd48c7c-lnpk5 4/4 Running 0 8d 172.25.27.26 172.25.27.26 <none> <none> service-02-ott-encoding-live-worker-7d74dddc79-d7mn7 4/4 Running 0 8d 172.25.27.26 172.25.27.26 <none> <none> service-03-enc-hd-encoding-live-worker-d47d78799-fgkjh 4/4 Running 9 21d 172.25.27.20 172.25.27.20 <none> <none> service-03-enc-sd-encoding-live-worker-5d56c8b844-6hsl6 4/4 Running 9 21d 172.25.27.26 172.25.27.26 <none> <none> service-03-iptv-hd-encoding-live-worker-57789d7567-llh9s 4/4 Running 0 8d 172.25.27.17 172.25.27.17 <none> <none> service-04-enc-hd-encoding-live-worker-67c7bff875-5gnbb 4/4 Running 9 21d 172.25.27.17 172.25.27.17 <none> <none> service-04-enc-sd-encoding-live-worker-6cd7f464d8-9kr8v 4/4 Running 9 21d 172.25.27.17 172.25.27.17 <none> <none> service-04-iptv-hd-encoding-live-worker-8d7646985-xnzwl 4/4 Running 0 8d 172.25.27.21 172.25.27.21 <none> <none> service-05-enc-hd-encoding-live-worker-7bff86bff-nmx4g 4/4 Running 9 21d 172.25.27.26 172.25.27.26 <none> <none> service-05-enc-sd-encoding-live-worker-75797fbd44-g2sv8 4/4 Running 9 21d 172.25.27.20 172.25.27.20 <none> <none> service-06-enc-hd-encoding-live-worker-5f8f87df9-bwdxr 4/4 Running 9 21d 172.25.27.17 172.25.27.17 <none> <none> service-06-enc-sd-encoding-live-worker-6c9b7f7fd4-g66q5 4/4 Running 10 21d 172.25.27.20 172.25.27.20 <none> <none> service-07-enc-hd-encoding-live-worker-57bfd9bff9-25sgv 4/4 Running 0 10d 172.25.27.21 172.25.27.21 <none> <none> service-08-enc-hd-encoding-live-worker-6cdf78b9c9-r5fbv 4/4 Running 1 10d 172.25.27.26 172.25.27.26 <none> <none> service-09-enc-hd-encoding-live-worker-84f7db7dd7-pfbxg 4/4 Running 1 10d 172.25.27.21 172.25.27.21 <none> <none> service-dtt01-1-enc-hd-encoding-live-worker-5958cddc97-gbczc 4/4 Running 0 9d 172.25.27.21 172.25.27.21 <none> <none> service-dtt01-2-enc-hd-encoding-live-worker-5bdfc9dc9-rfbcc 4/4 Running 0 9d 172.25.27.26 172.25.27.26 <none> <none> service-dtt01-3-enc-hd-encoding-live-worker-8696d7955c-gqwxk 4/4 Running 0 9d 172.25.27.21 172.25.27.21 <none> <none> service-dtt01-4-enc-hd-encoding-live-worker-85f95cd6c9-vswhx 4/4 Running 0 9d 172.25.27.17 172.25.27.17 <none> <none> service-dtt01-5-enc-hd-encoding-live-worker-65447cbdc7-swxt2 4/4 Running 0 9d 172.25.27.20 172.25.27.20 <none> <none> service-dtt02-1-enc-hd-encoding-live-worker-d8bdd8c4c-knwr8 4/4 Running 0 9d 172.25.27.21 172.25.27.21 <none> <none> service-dtt02-2-enc-hd-encoding-live-worker-5dd4f4d445-mdtkh 4/4 Running 0 9d 172.25.27.20 172.25.27.20 <none> <none> service-dtt02-3-enc-hd-encoding-live-worker-65d5fb7455-5slpw 4/4 Running 0 9d 172.25.27.26 172.25.27.26 <none> <none> service-dtt02-4-enc-hd-encoding-live-worker-549dbbcf8f-5dxlz 4/4 Running 0 9d 172.25.27.17 172.25.27.17 <none> <none> service-dtt02-5-enc-hd-encoding-live-worker-d678b88ff-pq66k 4/4 Running 0 9d 172.25.27.20 172.25.27.20 <none> <none> service-dtt02-6-enc-sd-encoding-live-worker-7844d9b7d4-2cnz8 4/4 Running 14 9d 172.25.27.26 172.25.27.26 <none> <none>
kubernetes_k8.1618507542.txt.gz · Last modified: 2023/03/09 22:35 (external edit)