# K8s rescue plan
··· Reset the virtual machine image, but did not reset my brain!
## cause
Directly power off all nodes and set Docker to not start up
## Can it be reproduced
Yes, the number of replicates is 2
##Analysis process
Restore the virtual machine to the normal state of the k8s cluster, observe the results of "Docker ps - a" and record it
### master
```shell
# masternode
[root@k8smaster ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
k8smaster Ready control-plane,master 7d21h v1.23.6
k8snode01 Ready <none> 7d21h v1.23.6
k8snode02 Ready <none> 7d21h v1.23.6
[root@k8smaster ~]#
[root@k8smaster ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c12f3abcf771 a4ca41631cc7 "/coredns -conf /etc…" 7 days ago Up 7 days k8s_coredns_coredns-6d8c4cb4d-6jjh5_kube-system_5880858e-7211-4bbb-8c93-90c928f68622_0
95edcbc13770 a4ca41631cc7 "/coredns -conf /etc…" 7 days ago Up 7 days k8s_coredns_coredns-6d8c4cb4d-d7st4_kube-system_078cc83a-b061-4e81-bcf6-cf21073cdd4e_0
5b0e34035658 registry.aliyuncs.com/google_containers/pause:3.6 "/pause" 7 days ago Up 7 days k8s_POD_coredns-6d8c4cb4d-6jjh5_kube-system_5880858e-7211-4bbb-8c93-90c928f68622_0
e78106ac1966 registry.aliyuncs.com/google_containers/pause:3.6 "/pause" 7 days ago Up 7 days k8s_POD_coredns-6d8c4cb4d-d7st4_kube-system_078cc83a-b061-4e81-bcf6-cf21073cdd4e_0
68aba3e09b9b d73868a08083 "/opt/bin/flanneld -…" 7 days ago Up 7 days k8s_kube-flannel_kube-flannel-ds-zz5cl_kube-flannel_24b1f309-0143-4627-8770-b78a7de603b0_0
95c152dd0190 registry.aliyuncs.com/google_containers/pause:3.6 "/pause" 7 days ago Up 7 days k8s_POD_kube-flannel-ds-zz5cl_kube-flannel_24b1f309-0143-4627-8770-b78a7de603b0_0
d011952c4be9 f21c8d21558c "/usr/local/bin/kube…" 7 days ago Up 7 days k8s_kube-proxy_kube-proxy-2g5ql_kube-system_5d508d75-495a-4159-87ee-20a29d9061e7_0
47da71089e8e registry.aliyuncs.com/google_containers/pause:3.6 "/pause" 7 days ago Up 7 days k8s_POD_kube-proxy-2g5ql_kube-system_5d508d75-495a-4159-87ee-20a29d9061e7_0
282fbf54321f bc6794cb54ac "kube-scheduler --au…" 7 days ago Up 7 days k8s_kube-scheduler_kube-scheduler-k8smaster_kube-system_c02a11b9d6ed102dae71c2a5ba394332_0
c650a0a82e8f 25f8c7f3da61 "etcd --advertise-cl…" 7 days ago Up 7 days k8s_etcd_etcd-k8smaster_kube-system_3b9b93edf9e5751a1bf3a1e147cce967_0
81b1b2d16921 1dab4fc7b6e0 "kube-controller-man…" 7 days ago Up 7 days k8s_kube-controller-manager_kube-controller-manager-k8smaster_kube-system_62f82849ccf351f48be952f05e36c490_0
3ddac11bfee8 62bc5d8258d6 "kube-apiserver --ad…" 7 days ago Up 7 days k8s_kube-apiserver_kube-apiserver-k8smaster_kube-system_ab1948dc99a13b01ea539c462d17491c_0
5b5d34542b36 registry.aliyuncs.com/google_containers/pause:3.6 "/pause" 7 days ago Up 7 days k8s_POD_kube-scheduler-k8smaster_kube-system_c02a11b9d6ed102dae71c2a5ba394332_0
a9bfe6ec50de registry.aliyuncs.com/google_containers/pause:3.6 "/pause" 7 days ago Up 7 days k8s_POD_kube-controller-manager-k8smaster_kube-system_62f82849ccf351f48be952f05e36c490_0
754226687c2f registry.aliyuncs.com/google_containers/pause:3.6 "/pause" 7 days ago Up 7 days k8s_POD_kube-apiserver-k8smaster_kube-system_ab1948dc99a13b01ea539c462d17491c_0
4d09932ad92f registry.aliyuncs.com/google_containers/pause:3.6 "/pause" 7 days ago Up 7 days k8s_POD_etcd-k8smaster_kube-system_3b9b93edf9e5751a1bf3a1e147cce967_0
[root@k8smaster ~]#
[root@k8smaster ~]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c12f3abcf771 a4ca41631cc7 "/coredns -conf /etc…" 7 days ago Up 7 days k8s_coredns_coredns-6d8c4cb4d-6jjh5_kube-system_5880858e-7211-4bbb-8c93-90c928f68622_0
95edcbc13770 a4ca41631cc7 "/coredns -conf /etc…" 7 days ago Up 7 days k8s_coredns_coredns-6d8c4cb4d-d7st4_kube-system_078cc83a-b061-4e81-bcf6-cf21073cdd4e_0
5b0e34035658 registry.aliyuncs.com/google_containers/pause:3.6 "/pause" 7 days ago Up 7 days k8s_POD_coredns-6d8c4cb4d-6jjh5_kube-system_5880858e-7211-4bbb-8c93-90c928f68622_0
e78106ac1966 registry.aliyuncs.com/google_containers/pause:3.6 "/pause" 7 days ago Up 7 days k8s_POD_coredns-6d8c4cb4d-d7st4_kube-system_078cc83a-b061-4e81-bcf6-cf21073cdd4e_0
68aba3e09b9b d73868a08083 "/opt/bin/flanneld -…" 7 days ago Up 7 days k8s_kube-flannel_kube-flannel-ds-zz5cl_kube-flannel_24b1f309-0143-4627-8770-b78a7de603b0_0
0936e9013449 flannel/flannel "cp -f /etc/kube-fla…" 7 days ago Exited (0) 7 days ago k8s_install-cni_kube-flannel-ds-zz5cl_kube-flannel_24b1f309-0143-4627-8770-b78a7de603b0_0
88cf1da60660 flannel/flannel-cni-plugin "cp -f /flannel /opt…" 7 days ago Exited (0) 7 days ago k8s_install-cni-plugin_kube-flannel-ds-zz5cl_kube-flannel_24b1f309-0143-4627-8770-b78a7de603b0_0
95c152dd0190 registry.aliyuncs.com/google_containers/pause:3.6 "/pause" 7 days ago Up 7 days k8s_POD_kube-flannel-ds-zz5cl_kube-flannel_24b1f309-0143-4627-8770-b78a7de603b0_0
d011952c4be9 f21c8d21558c "/usr/local/bin/kube…" 7 days ago Up 7 days k8s_kube-proxy_kube-proxy-2g5ql_kube-system_5d508d75-495a-4159-87ee-20a29d9061e7_0
47da71089e8e registry.aliyuncs.com/google_containers/pause:3.6 "/pause" 7 days ago Up 7 days k8s_POD_kube-proxy-2g5ql_kube-system_5d508d75-495a-4159-87ee-20a29d9061e7_0
282fbf54321f bc6794cb54ac "kube-scheduler --au…" 7 days ago Up 7 days k8s_kube-scheduler_kube-scheduler-k8smaster_kube-system_c02a11b9d6ed102dae71c2a5ba394332_0
c650a0a82e8f 25f8c7f3da61 "etcd --advertise-cl…" 7 days ago Up 7 days k8s_etcd_etcd-k8smaster_kube-system_3b9b93edf9e5751a1bf3a1e147cce967_0
81b1b2d16921 1dab4fc7b6e0 "kube-controller-man…" 7 days ago Up 7 days k8s_kube-controller-manager_kube-controller-manager-k8smaster_kube-system_62f82849ccf351f48be952f05e36c490_0
3ddac11bfee8 62bc5d8258d6 "kube-apiserver --ad…" 7 days ago Up 7 days k8s_kube-apiserver_kube-apiserver-k8smaster_kube-system_ab1948dc99a13b01ea539c462d17491c_0
5b5d34542b36 registry.aliyuncs.com/google_containers/pause:3.6 "/pause" 7 days ago Up 7 days k8s_POD_kube-scheduler-k8smaster_kube-system_c02a11b9d6ed102dae71c2a5ba394332_0
a9bfe6ec50de registry.aliyuncs.com/google_containers/pause:3.6 "/pause" 7 days ago Up 7 days k8s_POD_kube-controller-manager-k8smaster_kube-system_62f82849ccf351f48be952f05e36c490_0
754226687c2f registry.aliyuncs.com/google_containers/pause:3.6 "/pause" 7 days ago Up 7 days k8s_POD_kube-apiserver-k8smaster_kube-system_ab1948dc99a13b01ea539c462d17491c_0
4d09932ad92f registry.aliyuncs.com/google_containers/pause:3.6 "/pause" 7 days ago Up 7 days k8s_POD_etcd-k8smaster_kube-system_3b9b93edf9e5751a1bf3a1e147cce967_0
[root@k8smaster ~]#
[root@k8smaster ~]# docker ps -a | grep Exited
0936e9013449 flannel/flannel "cp -f /etc/kube-fla…" 7 days ago Exited (0) 7 days ago k8s_install-cni_kube-flannel-ds-zz5cl_kube-flannel_24b1f309-0143-4627-8770-b78a7de603b0_0
88cf1da60660 flannel/flannel-cni-plugin "cp -f /flannel /opt…" 7 days ago Exited (0) 7 days ago k8s_install-cni-plugin_kube-flannel-ds-zz5cl_kube-flannel_24b1f309-0143-4627-8770-b78a7de603b0_0
[root@k8smaster ~]#
[root@k8smaster ~]# kubectl get all
NAME READY STATUS RESTARTS AGE
pod/k8s-nginx-6d779d947c-59qh8 1/1 Running 0 7d21h
pod/k8s-nginx-6d779d947c-djzx7 1/1 Running 0 7d21h
pod/k8s-nginx-6d779d947c-wxhx2 1/1 Running 0 7d21h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.1.0.1 <none> 443/TCP 7d21h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/k8s-nginx 3/3 3 3 7d21h
NAME DESIRED CURRENT READY AGE
replicaset.apps/k8s-nginx-6d779d947c 3 3 3 7d21h
[root@k8smaster ~]#
[root@k8smaster ~]# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /usr/lib/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Mon 2023-09-11 16:15:43 CST; 1 weeks 0 days ago
Docs: https://kubernetes.io/docs/
Main PID: 13497 (kubelet)
Tasks: 15
Memory: 68.3M
CGroup: /system.slice/kubelet.service
└─13497 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --network-plugin=cni --po...
Sep 11 16:24:44 k8smaster kubelet[13497]: I0911 16:24:44.189915 13497 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes...
Sep 11 16:24:44 k8smaster kubelet[13497]: I0911 16:24:44.189968 13497 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrpx5\" (UniqueName: \"ku...
Sep 11 16:24:44 k8smaster kubelet[13497]: I0911 16:24:44.190014 13497 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmkws\" (UniqueName: \"ku...
Sep 11 16:24:44 k8smaster kubelet[13497]: I0911 16:24:44.190036 13497 reconciler.go:221] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes...
Sep 11 16:24:44 k8smaster kubelet[13497]: I0911 16:24:44.711527 13497 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="e78106ac1966087346b4b511a425ac7a3...6b1130cab1b65"
Sep 11 16:24:44 k8smaster kubelet[13497]: map[string]interface {}{"cniVersion":"0.3.1", "hairpinMode":true, "ipMasq":false, "ipam":map[string]interface {}{"ranges":[][]map[string]interface {}{[]map[strin...
Sep 11 16:24:44 k8smaster kubelet[13497]: delegateAdd: netconf sent to delegate plugin:
Sep 11 16:24:44 k8smaster kubelet[13497]: {"cniVersion":"0.3.1","hairpinMode":true,"ipMasq":false,"ipam":{"ranges":[[{"subnet":"10.244.0.0/24"}]],"routes":[{"dst":"10.244.0.0/16"}],"type":"...ype":"bridge"}
Sep 11 16:24:44 k8smaster kubelet[13497]: map[string]interface {}{"cniVersion":"0.3.1", "hairpinMode":true, "ipMasq":false, "ipam":map[string]interface {}{"ranges":[][]map[string]interface {}{[]map[strin...
Sep 11 16:24:44 k8smaster kubelet[13497]: delegateAdd: netconf sent to delegate plugin:
Hint: Some lines were ellipsized, use -l to show in full.
[root@k8smaster ~]#
[root@k8smaster ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:37:27:88 brd ff:ff:ff:ff:ff:ff
inet 192.168.7.100/24 brd 192.168.7.255 scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe37:2788/64 scope link noprefixroute
valid_lft forever preferred_lft forever
3: ens36: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:37:27:92 brd ff:ff:ff:ff:ff:ff
inet 192.168.157.149/24 brd 192.168.157.255 scope global noprefixroute dynamic ens36
valid_lft 1671sec preferred_lft 1671sec
inet6 fe80::bdcf:3ffc:b6f6:f1ed/64 scope link noprefixroute
valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:e0:e8:64:12 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
5: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
link/ether f2:94:58:2b:80:92 brd ff:ff:ff:ff:ff:ff
inet 10.244.0.0/32 scope global flannel.1
valid_lft forever preferred_lft forever
inet6 fe80::f094:58ff:fe2b:8092/64 scope link
valid_lft forever preferred_lft forever
6: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
link/ether 4e:41:5e:15:b7:62 brd ff:ff:ff:ff:ff:ff
inet 10.244.0.1/24 brd 10.244.0.255 scope global cni0
valid_lft forever preferred_lft forever
inet6 fe80::4c41:5eff:fe15:b762/64 scope link
valid_lft forever preferred_lft forever
7: vetha9ed6409@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
link/ether 76:e0:63:90:81:7d brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::74e0:63ff:fe90:817d/64 scope link
valid_lft forever preferred_lft forever
8: veth7a07689e@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
link/ether c6:66:fb:c0:75:cd brd ff:ff:ff:ff:ff:ff link-netnsid 1
inet6 fe80::c466:fbff:fec0:75cd/64 scope link
valid_lft forever preferred_lft forever
[root@k8smaster ~]#
[root@k8smaster ~]# route
-bash: route: command not found
[root@k8smaster ~]#
[root@k8smaster ~]# iptables -L -n
Chain INPUT (policy ACCEPT)
target prot opt source destination
KUBE-NODEPORTS all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes health check service ports */
KUBE-EXTERNAL-SERVICES all -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* kubernetes externally-visible service portals */
KUBE-FIREWALL all -- 0.0.0.0/0 0.0.0.0/0
Chain FORWARD (policy ACCEPT)
target prot opt source destination
KUBE-FORWARD all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes forwarding rules */
KUBE-SERVICES all -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* kubernetes service portals */
KUBE-EXTERNAL-SERVICES all -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* kubernetes externally-visible service portals */
DOCKER-USER all -- 0.0.0.0/0 0.0.0.0/0
DOCKER-ISOLATION-STAGE-1 all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
DOCKER all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
FLANNEL-FWD all -- 0.0.0.0/0 0.0.0.0/0 /* flanneld forward */
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
KUBE-SERVICES all -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* kubernetes service portals */
KUBE-FIREWALL all -- 0.0.0.0/0 0.0.0.0/0
Chain DOCKER (1 references)
target prot opt source destination
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target prot opt source destination
DOCKER-ISOLATION-STAGE-2 all -- 0.0.0.0/0 0.0.0.0/0
RETURN all -- 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target prot opt source destination
DROP all -- 0.0.0.0/0 0.0.0.0/0
RETURN all -- 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-USER (1 references)
target prot opt source destination
RETURN all -- 0.0.0.0/0 0.0.0.0/0
Chain FLANNEL-FWD (1 references)
target prot opt source destination
ACCEPT all -- 10.244.0.0/16 0.0.0.0/0 /* flanneld forward */
ACCEPT all -- 0.0.0.0/0 10.244.0.0/16 /* flanneld forward */
Chain KUBE-EXTERNAL-SERVICES (2 references)
target prot opt source destination
Chain KUBE-FIREWALL (2 references)
target prot opt source destination
DROP all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000
DROP all -- !127.0.0.0/8 127.0.0.0/8 /* block incoming localnet connections */ ! ctstate RELATED,ESTABLISHED,DNAT
Chain KUBE-FORWARD (1 references)
target prot opt source destination
DROP all -- 0.0.0.0/0 0.0.0.0/0 ctstate INVALID
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes forwarding rules */ mark match 0x4000/0x4000
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes forwarding conntrack rule */ ctstate RELATED,ESTABLISHED
Chain KUBE-KUBELET-CANARY (0 references)
target prot opt source destination
Chain KUBE-NODEPORTS (1 references)
target prot opt source destination
Chain KUBE-PROXY-CANARY (0 references)
target prot opt source destination
Chain KUBE-SERVICES (2 references)
target prot opt source destination
[root@k8smaster ~]#
```
Analysis shows that in a normal cluster, k8s can see that there are two initialization containers in the node node that are generated by copying files, while the others are all in the up state
### node01
```shell
#node01
[root@k8snode01 ~]# kubectl get node
The connection to the server localhost:8080 was refused - did you specify the right host or port?
[root@k8snode01 ~]#
[root@k8snode01 ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
02d1b7ed4d94 nginx "/docker-entrypoint.…" 7 days ago Up 7 days k8s_nginx_k8s-nginx-6d779d947c-59qh8_default_056c7a0e-ac54-40f0-ad4d-70d5ec6f2a92_0
ad3c063c2818 registry.aliyuncs.com/google_containers/pause:3.6 "/pause" 7 days ago Up 7 days k8s_POD_k8s-nginx-6d779d947c-59qh8_default_056c7a0e-ac54-40f0-ad4d-70d5ec6f2a92_0
df414c590b55 d73868a08083 "/opt/bin/flanneld -…" 7 days ago Up 7 days k8s_kube-flannel_kube-flannel-ds-2bf4k_kube-flannel_5e1afce5-b4d3-45fd-9c58-4cace2b43283_0
25d0c45d1b2d registry.aliyuncs.com/google_containers/kube-proxy "/usr/local/bin/kube…" 7 days ago Up 7 days k8s_kube-proxy_kube-proxy-zr2m6_kube-system_7e2bdbc0-11ee-4263-8752-5342d93e9c5f_0
f426822aee9e registry.aliyuncs.com/google_containers/pause:3.6 "/pause" 7 days ago Up 7 days k8s_POD_kube-proxy-zr2m6_kube-system_7e2bdbc0-11ee-4263-8752-5342d93e9c5f_0
f8f25a24e78f registry.aliyuncs.com/google_containers/pause:3.6 "/pause" 7 days ago Up 7 days k8s_POD_kube-flannel-ds-2bf4k_kube-flannel_5e1afce5-b4d3-45fd-9c58-4cace2b43283_0
[root@k8snode01 ~]#
[root@k8snode01 ~]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
02d1b7ed4d94 nginx "/docker-entrypoint.…" 7 days ago Up 7 days k8s_nginx_k8s-nginx-6d779d947c-59qh8_default_056c7a0e-ac54-40f0-ad4d-70d5ec6f2a92_0
ad3c063c2818 registry.aliyuncs.com/google_containers/pause:3.6 "/pause" 7 days ago Up 7 days k8s_POD_k8s-nginx-6d779d947c-59qh8_default_056c7a0e-ac54-40f0-ad4d-70d5ec6f2a92_0
df414c590b55 d73868a08083 "/opt/bin/flanneld -…" 7 days ago Up 7 days k8s_kube-flannel_kube-flannel-ds-2bf4k_kube-flannel_5e1afce5-b4d3-45fd-9c58-4cace2b43283_0
70d322520704 flannel/flannel "cp -f /etc/kube-fla…" 7 days ago Exited (0) 7 days ago k8s_install-cni_kube-flannel-ds-2bf4k_kube-flannel_5e1afce5-b4d3-45fd-9c58-4cace2b43283_0
abac9835c972 flannel/flannel-cni-plugin "cp -f /flannel /opt…" 7 days ago Exited (0) 7 days ago k8s_install-cni-plugin_kube-flannel-ds-2bf4k_kube-flannel_5e1afce5-b4d3-45fd-9c58-4cace2b43283_0
25d0c45d1b2d registry.aliyuncs.com/google_containers/kube-proxy "/usr/local/bin/kube…" 7 days ago Up 7 days k8s_kube-proxy_kube-proxy-zr2m6_kube-system_7e2bdbc0-11ee-4263-8752-5342d93e9c5f_0
f426822aee9e registry.aliyuncs.com/google_containers/pause:3.6 "/pause" 7 days ago Up 7 days k8s_POD_kube-proxy-zr2m6_kube-system_7e2bdbc0-11ee-4263-8752-5342d93e9c5f_0
f8f25a24e78f registry.aliyuncs.com/google_containers/pause:3.6 "/pause" 7 days ago Up 7 days k8s_POD_kube-flannel-ds-2bf4k_kube-flannel_5e1afce5-b4d3-45fd-9c58-4cace2b43283_0
[root@k8snode01 ~]#
[root@k8snode01 ~]# docker ps -a | grep Exited
70d322520704 flannel/flannel "cp -f /etc/kube-fla…" 7 days ago Exited (0) 7 days ago k8s_install-cni_kube-flannel-ds-2bf4k_kube-flannel_5e1afce5-b4d3-45fd-9c58-4cace2b43283_0
abac9835c972 flannel/flannel-cni-plugin "cp -f /flannel /opt…" 7 days ago Exited (0) 7 days ago k8s_install-cni-plugin_kube-flannel-ds-2bf4k_kube-flannel_5e1afce5-b4d3-45fd-9c58-4cace2b43283_0
[root@k8snode01 ~]#
[root@k8snode01 ~]# route
-bash: route: command not found
[root@k8snode01 ~]#
[root@k8snode01 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:06:ce:c8 brd ff:ff:ff:ff:ff:ff
inet 192.168.7.101/24 brd 192.168.7.255 scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe06:cec8/64 scope link noprefixroute
valid_lft forever preferred_lft forever
3: ens36: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:06:ce:d2 brd ff:ff:ff:ff:ff:ff
inet 192.168.157.150/24 brd 192.168.157.255 scope global noprefixroute dynamic ens36
valid_lft 1083sec preferred_lft 1083sec
inet6 fe80::2bba:1bc9:5ff2:c21a/64 scope link noprefixroute
valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:7b:05:53:6f brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
5: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
link/ether 7a:b7:34:d3:fa:f1 brd ff:ff:ff:ff:ff:ff
inet 10.244.1.0/32 scope global flannel.1
valid_lft forever preferred_lft forever
inet6 fe80::78b7:34ff:fed3:faf1/64 scope link
valid_lft forever preferred_lft forever
6: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
link/ether c6:41:f3:f6:6b:39 brd ff:ff:ff:ff:ff:ff
inet 10.244.1.1/24 brd 10.244.1.255 scope global cni0
valid_lft forever preferred_lft forever
inet6 fe80::c441:f3ff:fef6:6b39/64 scope link
valid_lft forever preferred_lft forever
7: vethdffd1800@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
link/ether 6e:f9:14:ca:b7:3a brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::6cf9:14ff:feca:b73a/64 scope link
valid_lft forever preferred_lft forever
[root@k8snode01 ~]#
[root@k8snode01 ~]# iptables -L -n
Chain INPUT (policy ACCEPT)
target prot opt source destination
KUBE-NODEPORTS all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes health check service ports */
KUBE-EXTERNAL-SERVICES all -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* kubernetes externally-visible service portals */
KUBE-FIREWALL all -- 0.0.0.0/0 0.0.0.0/0
Chain FORWARD (policy ACCEPT)
target prot opt source destination
KUBE-FORWARD all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes forwarding rules */
KUBE-SERVICES all -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* kubernetes service portals */
KUBE-EXTERNAL-SERVICES all -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* kubernetes externally-visible service portals */
DOCKER-USER all -- 0.0.0.0/0 0.0.0.0/0
DOCKER-ISOLATION-STAGE-1 all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
DOCKER all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
FLANNEL-FWD all -- 0.0.0.0/0 0.0.0.0/0 /* flanneld forward */
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
KUBE-SERVICES all -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* kubernetes service portals */
KUBE-FIREWALL all -- 0.0.0.0/0 0.0.0.0/0
Chain DOCKER (1 references)
target prot opt source destination
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target prot opt source destination
DOCKER-ISOLATION-STAGE-2 all -- 0.0.0.0/0 0.0.0.0/0
RETURN all -- 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target prot opt source destination
DROP all -- 0.0.0.0/0 0.0.0.0/0
RETURN all -- 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-USER (1 references)
target prot opt source destination
RETURN all -- 0.0.0.0/0 0.0.0.0/0
Chain FLANNEL-FWD (1 references)
target prot opt source destination
ACCEPT all -- 10.244.0.0/16 0.0.0.0/0 /* flanneld forward */
ACCEPT all -- 0.0.0.0/0 10.244.0.0/16 /* flanneld forward */
Chain KUBE-EXTERNAL-SERVICES (2 references)
target prot opt source destination
Chain KUBE-FIREWALL (2 references)
target prot opt source destination
DROP all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000
DROP all -- !127.0.0.0/8 127.0.0.0/8 /* block incoming localnet connections */ ! ctstate RELATED,ESTABLISHED,DNAT
Chain KUBE-FORWARD (1 references)
target prot opt source destination
DROP all -- 0.0.0.0/0 0.0.0.0/0 ctstate INVALID
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes forwarding rules */ mark match 0x4000/0x4000
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes forwarding conntrack rule */ ctstate RELATED,ESTABLISHED
Chain KUBE-KUBELET-CANARY (0 references)
target prot opt source destination
Chain KUBE-NODEPORTS (1 references)
target prot opt source destination
Chain KUBE-PROXY-CANARY (0 references)
target prot opt source destination
Chain KUBE-SERVICES (2 references)
target prot opt source destination
[root@k8snode01 ~]#
```
Analysis shows that in a normal cluster, k8s can see that there are two initialization containers in the node node that are generated by copying files, while the others are all in the up state
### node02
```shell
#node02
[root@k8snode02 ~]# kubectl get node
The connection to the server localhost:8080 was refused - did you specify the right host or port?
[root@k8snode02 ~]#
[root@k8snode02 ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3e4b9444378b nginx "/docker-entrypoint.…" 7 days ago Up 7 days k8s_nginx_k8s-nginx-6d779d947c-wxhx2_default_43a547de-284a-4055-879b-7ecd926d0a41_0
25593b6e2880 nginx "/docker-entrypoint.…" 7 days ago Up 7 days k8s_nginx_k8s-nginx-6d779d947c-djzx7_default_01b1b881-d996-478c-bf7d-4f9491d0d63f_0
c4bba5dbdfb3 registry.aliyuncs.com/google_containers/pause:3.6 "/pause" 7 days ago Up 7 days k8s_POD_k8s-nginx-6d779d947c-djzx7_default_01b1b881-d996-478c-bf7d-4f9491d0d63f_0
d42203824011 registry.aliyuncs.com/google_containers/pause:3.6 "/pause" 7 days ago Up 7 days k8s_POD_k8s-nginx-6d779d947c-wxhx2_default_43a547de-284a-4055-879b-7ecd926d0a41_0
befe5b6209fd d73868a08083 "/opt/bin/flanneld -…" 7 days ago Up 7 days k8s_kube-flannel_kube-flannel-ds-gpnpx_kube-flannel_c6a1656c-d0cf-4026-b0d7-ff8885f24c1a_0
eb8137f30356 registry.aliyuncs.com/google_containers/kube-proxy "/usr/local/bin/kube…" 7 days ago Up 7 days k8s_kube-proxy_kube-proxy-n95s6_kube-system_dee6cae8-47a8-4b79-8eb9-1aeea3f673dd_0
9de41647ef33 registry.aliyuncs.com/google_containers/pause:3.6 "/pause" 7 days ago Up 7 days k8s_POD_kube-flannel-ds-gpnpx_kube-flannel_c6a1656c-d0cf-4026-b0d7-ff8885f24c1a_0
fa5300178a42 registry.aliyuncs.com/google_containers/pause:3.6 "/pause" 7 days ago Up 7 days k8s_POD_kube-proxy-n95s6_kube-system_dee6cae8-47a8-4b79-8eb9-1aeea3f673dd_0
[root@k8snode02 ~]#
[root@k8snode02 ~]#
[root@k8snode02 ~]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3e4b9444378b nginx "/docker-entrypoint.…" 7 days ago Up 7 days k8s_nginx_k8s-nginx-6d779d947c-wxhx2_default_43a547de-284a-4055-879b-7ecd926d0a41_0
25593b6e2880 nginx "/docker-entrypoint.…" 7 days ago Up 7 days k8s_nginx_k8s-nginx-6d779d947c-djzx7_default_01b1b881-d996-478c-bf7d-4f9491d0d63f_0
c4bba5dbdfb3 registry.aliyuncs.com/google_containers/pause:3.6 "/pause" 7 days ago Up 7 days k8s_POD_k8s-nginx-6d779d947c-djzx7_default_01b1b881-d996-478c-bf7d-4f9491d0d63f_0
d42203824011 registry.aliyuncs.com/google_containers/pause:3.6 "/pause" 7 days ago Up 7 days k8s_POD_k8s-nginx-6d779d947c-wxhx2_default_43a547de-284a-4055-879b-7ecd926d0a41_0
befe5b6209fd d73868a08083 "/opt/bin/flanneld -…" 7 days ago Up 7 days k8s_kube-flannel_kube-flannel-ds-gpnpx_kube-flannel_c6a1656c-d0cf-4026-b0d7-ff8885f24c1a_0
80713a3f68d4 flannel/flannel "cp -f /etc/kube-fla…" 7 days ago Exited (0) 7 days ago k8s_install-cni_kube-flannel-ds-gpnpx_kube-flannel_c6a1656c-d0cf-4026-b0d7-ff8885f24c1a_0
354424bed696 flannel/flannel-cni-plugin "cp -f /flannel /opt…" 7 days ago Exited (0) 7 days ago k8s_install-cni-plugin_kube-flannel-ds-gpnpx_kube-flannel_c6a1656c-d0cf-4026-b0d7-ff8885f24c1a_0
eb8137f30356 registry.aliyuncs.com/google_containers/kube-proxy "/usr/local/bin/kube…" 7 days ago Up 7 days k8s_kube-proxy_kube-proxy-n95s6_kube-system_dee6cae8-47a8-4b79-8eb9-1aeea3f673dd_0
9de41647ef33 registry.aliyuncs.com/google_containers/pause:3.6 "/pause" 7 days ago Up 7 days k8s_POD_kube-flannel-ds-gpnpx_kube-flannel_c6a1656c-d0cf-4026-b0d7-ff8885f24c1a_0
fa5300178a42 registry.aliyuncs.com/google_containers/pause:3.6 "/pause" 7 days ago Up 7 days k8s_POD_kube-proxy-n95s6_kube-system_dee6cae8-47a8-4b79-8eb9-1aeea3f673dd_0
[root@k8snode02 ~]#
[root@k8snode02 ~]#
[root@k8snode02 ~]# docker ps -a | grep Exited
80713a3f68d4 flannel/flannel "cp -f /etc/kube-fla…" 7 days ago Exited (0) 7 days ago k8s_install-cni_kube-flannel-ds-gpnpx_kube-flannel_c6a1656c-d0cf-4026-b0d7-ff8885f24c1a_0
354424bed696 flannel/flannel-cni-plugin "cp -f /flannel /opt…" 7 days ago Exited (0) 7 days ago k8s_install-cni-plugin_kube-flannel-ds-gpnpx_kube-flannel_c6a1656c-d0cf-4026-b0d7-ff8885f24c1a_0
[root@k8snode02 ~]#
[root@k8snode02 ~]#
[root@k8snode02 ~]# route
-bash: route: command not found
[root@k8snode02 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:a4:87:a1 brd ff:ff:ff:ff:ff:ff
inet 192.168.7.102/24 brd 192.168.7.255 scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fea4:87a1/64 scope link noprefixroute
valid_lft forever preferred_lft forever
3: ens36: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:a4:87:ab brd ff:ff:ff:ff:ff:ff
inet 192.168.157.148/24 brd 192.168.157.255 scope global noprefixroute dynamic ens36
valid_lft 1193sec preferred_lft 1193sec
inet6 fe80::ba59:a533:42da:cbfc/64 scope link noprefixroute
valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:e1:ee:f1:b8 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
5: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
link/ether fa:c3:b3:14:5d:1b brd ff:ff:ff:ff:ff:ff
inet 10.244.2.0/32 scope global flannel.1
valid_lft forever preferred_lft forever
inet6 fe80::f8c3:b3ff:fe14:5d1b/64 scope link
valid_lft forever preferred_lft forever
6: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
link/ether aa:6c:34:ed:3b:28 brd ff:ff:ff:ff:ff:ff
inet 10.244.2.1/24 brd 10.244.2.255 scope global cni0
valid_lft forever preferred_lft forever
inet6 fe80::a86c:34ff:feed:3b28/64 scope link
valid_lft forever preferred_lft forever
7: veth3f1fc914@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
link/ether 12:74:54:3c:ef:77 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::1074:54ff:fe3c:ef77/64 scope link
valid_lft forever preferred_lft forever
8: veth1b04c2fb@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
link/ether 82:01:af:6c:ef:d9 brd ff:ff:ff:ff:ff:ff link-netnsid 1
inet6 fe80::8001:afff:fe6c:efd9/64 scope link
valid_lft forever preferred_lft forever
[root@k8snode02 ~]#
[root@k8snode02 ~]# iptables -L -n
Chain INPUT (policy ACCEPT)
target prot opt source destination
KUBE-NODEPORTS all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes health check service ports */
KUBE-EXTERNAL-SERVICES all -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* kubernetes externally-visible service portals */
KUBE-FIREWALL all -- 0.0.0.0/0 0.0.0.0/0
Chain FORWARD (policy ACCEPT)
target prot opt source destination
KUBE-FORWARD all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes forwarding rules */
KUBE-SERVICES all -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* kubernetes service portals */
KUBE-EXTERNAL-SERVICES all -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* kubernetes externally-visible service portals */
DOCKER-USER all -- 0.0.0.0/0 0.0.0.0/0
DOCKER-ISOLATION-STAGE-1 all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
DOCKER all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
FLANNEL-FWD all -- 0.0.0.0/0 0.0.0.0/0 /* flanneld forward */
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
KUBE-SERVICES all -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* kubernetes service portals */
KUBE-FIREWALL all -- 0.0.0.0/0 0.0.0.0/0
Chain DOCKER (1 references)
target prot opt source destination
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target prot opt source destination
DOCKER-ISOLATION-STAGE-2 all -- 0.0.0.0/0 0.0.0.0/0
RETURN all -- 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target prot opt source destination
DROP all -- 0.0.0.0/0 0.0.0.0/0
RETURN all -- 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-USER (1 references)
target prot opt source destination
RETURN all -- 0.0.0.0/0 0.0.0.0/0
Chain FLANNEL-FWD (1 references)
target prot opt source destination
ACCEPT all -- 10.244.0.0/16 0.0.0.0/0 /* flanneld forward */
ACCEPT all -- 0.0.0.0/0 10.244.0.0/16 /* flanneld forward */
Chain KUBE-EXTERNAL-SERVICES (2 references)
target prot opt source destination
Chain KUBE-FIREWALL (2 references)
target prot opt source destination
DROP all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000
DROP all -- !127.0.0.0/8 127.0.0.0/8 /* block incoming localnet connections */ ! ctstate RELATED,ESTABLISHED,DNAT
Chain KUBE-FORWARD (1 references)
target prot opt source destination
DROP all -- 0.0.0.0/0 0.0.0.0/0 ctstate INVALID
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes forwarding rules */ mark match 0x4000/0x4000
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes forwarding conntrack rule */ ctstate RELATED,ESTABLISHED
Chain KUBE-KUBELET-CANARY (0 references)
target prot opt source destination
Chain KUBE-NODEPORTS (1 references)
target prot opt source destination
Chain KUBE-PROXY-CANARY (0 references)
target prot opt source destination
Chain KUBE-SERVICES (2 references)
target prot opt source destination
[root@k8snode02 ~]#
```
Analysis shows that in a normal cluster, k8s can see that there are two initialization containers in the node node that are generated by copying files, while the others are all in the up state
## Recurrence of destructive operations
Directly turn off the virtual machine power!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
## Fault viewing
### master
```powershell
[root@k8smaster ~]# docker ps
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
[root@k8smaster ~]# ip a
具体见：https://github.com/heike-07/K8s-study 或 https://gitee.com/heike07code/K8s-study
0.1-K8s rescue plan.md 部分~，篇幅过长不展示。
另外支持作者程序：
一个用于mysql备份的整合 star~
