安裝Kuberentes(K8s)

發表於 分類於 ，

使用環境

Ubuntu 20.04
docker 20.10.7
kubernetes 1.24.3
kubectl 1.24.3
kustomize 4.54
kubeadm 1.24.3
kubelet 1.24.3

docker安裝(建議這邊不要使用docker desktop版本)

先移除舊版
sudo apt-get remove docker docker-engine docker.io containerd runc

更新現有apt資料庫(這一步跟我前面寫的docker桌面版本安裝一樣就不一一說明了)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
apt-get update
apt-get install apt-transport-https ca-certificates curl gnupg lsb-release

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

echo \  "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

apt-get update
apt-get install docker-ce docker-ce-cli containerd.io

apt-cache madison docker-ce

sudo apt-get install docker-ce=5:20.10.16~3-0~ubuntu-jammy  docker-ce-cli=5:20.10.16~3-0~ubuntu-jammy containerd.io

docker -v

Kuberentes安裝

在安裝 k8s 前，必須把系統上的 swap disable 。

1
2
3
4
sudo swapoff -a  #暫時性

sudo vim /etc/fstab  #永久的
# /swapfile ... ...
1
2
3
4
5
6
sudo apt-get update && sudo apt-get install -y apt-transport-https curl
curl -O https://packages.cloud.google.com/apt/doc/apt-key.gpg
sudo apt-key add apt-key.gpg
sudo apt-add-repository "deb http://apt.kubernetes.io/ kubernetes-xenial main"
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl

Kuberentes 在 master 中初始化 kubeadm(只有master端要做，node端請跳過這步)

1
sudo kubeadm init   --pod-network-cidr=10.244.0.0/16 --service-cidr=10.245.0.0/16 --apiserver-advertise-address=<master_IP>

重製kubeadm(有出問題再使用，使用完在重作上一步)

1
kubeadm reset -f

成功的話會看到下面顯示這個敘述

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.0.1:6443 --token f9bvtp.zfurci0tw593y4bu \
    --discovery-token-ca-cert-hash sha256:581d22b40c315129632cba6f16508dd82a6f5fc0d2ed9492391dddf00ae4af50

如果這裡出錯可能試試看

1
2
rm /etc/containerd/config.toml 
sudo systemctl restart containerd

權限調整

1
2
3
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Kuberentes檢查版本

1
2
3
kubectl version
kubeadm version
kubelet --version

Kuberentes檢查node的Status

1
2
3
4
kubectl get nodes
kubectl get pods -n kube-system
kubectl -n kube-system get pod -w

安裝CNI

這裡使用官方建議的Calico

curl https://projectcalico.docs.tigera.io/manifests/calico.yaml -O

注意: 在 apply calico.yaml 前，如果你有意變更 Pod CIDR 的 IP 的話，請修正3644行的指令，將其填上你 init 的 Pod CIDR IP 修改CALICO_IPV4POOL_CIDR
value:

kubectl apply -f calico.yaml

將node端加入cluster

查詢節點token(如果剛剛master端那步沒有記下來的話)

kubeadm token create --print-join-command 查完會出現類似下面Apply cluster node的指令 ### Apply cluster node

1
2
3
sudo kubeadm join 192.168.0.1:6443 --token f9bvtp.zfurci0tw593y4bu \
   --discovery-token-ca-cert-hash sha256:581d22b40c315129632cba6f16508dd82a6f5fc0d2ed9492391dddf00ae4af50

Kuberentes檢查元件狀態

kubectl -n kube-system get all

安裝Kuberentes dashboard

1
2
3
4
5
6
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.6.0/aio/deploy/recommended.yaml


kubectl proxy(要有節點加入才不會報錯)

kubectl describe pod kubernetes-dashboard-5947dc95db-n42cs -n kubernetes-dashboard(查詢dashboard狀態後面流水號會不同)

創建Kuberentes dashboard帳號

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
kubectl apply -f dashboard-adminuser.yaml

dashboard-adminuser.yaml內容如下
—---------------------------------------------------------
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kubernetes-dashboard
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kubernetes-dashboard
—-----------------------------------------------------

kubectl create sa admin-user

kubectl get sa
kubectl get secret
kubectl create token admin-user
kubectl create token admin-user --duration=999999h

刪除帳號

1
2
3
4
5
6
7
8
9
10
kubectl delete sa cicd
kubectl get sa,secret # all gone

sudo kubectl create clusterrolebinding  dashboard-cluster-admin --clusterrole=cluster-admin --serviceaccount=kubernetes-dashboard:dashboard-admin


kubectl delete clusterrolebinding kubernetes-dashboard
kubectl create clusterrolebinding kubernetes-dashboard --clusterrole=cluster-admin --serviceaccount=kube-system:kubernetes-dashboard --user=clusterUser
kubectl describe clusterrolebinding kubernetes-dashboard
https://kubernetes.io/docs/reference/access-authn-authz/rbac/

安裝kubernetes metrics-server(要安裝metrics才能在dashboard看到資料)

抱歉我累了...
請自行閱讀

https://github.com/kubernetes-sigs/metrics-server#readme

錯誤排除

  1. 遇到xxx.xxx.xxx.xxx:8080
    代表kubeadm inti後面少做權限調整

  2. 遇到xxx.xxx.xxx.xxx:6443

    1
    2
    3
    4
    sudo -i 
    swapoff -a 
    exit 
    strace -eopenat kubectl version

  3. 遇到kubernetes dashboard：connect : no route to host
    代表IPtable 出包了

    1
    2
    3
    4
    5
    6
    systemctl stop kubelet
    systemctl stop docker
    iptables - -flush
    iptables -tnat - - flush
    systemctl start kubelet
    systemctl start docker

  4. 其他故障排除

    1
    2
    3
    sudo systemctl restart docker
    sudo systemctl daemon-reload
    sudo systemctl restart kubelet

參考資料

  1. https://kangjw.me/ubuntu-20-04-%E5%AE%89%E8%A3%9Ddocker/
  2. https://www.gundam.com.tw/2021/11/04/%E5%A6%82%E4%BD%95%E5%9C%A8-ubuntu-20-04-%E4%B8%8A%E5%AE%89%E8%A3%9D%E5%92%8C%E4%BD%BF%E7%94%A8-docker/
  3. https://hackmd.io/@cnsrl/BkjmFubTw#Step-4-Install-CNI
  4. https://github.com/z416352/Kubeadm-installation
  5. https://blog.csdn.net/qq_43580215/article/details/125153959
  6. https://www.akiicat.com/2019/04/26/Kubernetes/kubernetes-reinstall-on-ubuntu/
  7. https://blog.csdn.net/woay2008/article/details/93250137
  8. https://itnext.io/big-change-in-k8s-1-24-about-serviceaccounts-and-their-secrets-4b909a4af4e0
  9. https://kubernetes.io/zh-cn/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/
  10. https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#pod-network
  11. https://ithelp.ithome.com.tw/articles/10203675
  12. https://github.com/kubernetes/kubernetes/issues/53533
  13. https://projectcalico.docs.tigera.io/getting-started/kubernetes/self-managed-onprem/onpremises#install-calico-with-etcd-datastore